]> Swisscom

Binzring 17 Zurich 8045 CH thomas.graf@swisscom.com

Swisscom

Binzring 17 Zurich 8045 CH ahmed.elhassany@swisscom.com

INSA-Lyon

Lyon FR alex.huang-feng@insa-lyon.fr

Everything OPS

Liege BE benoit@everything-ops.net

NTT

Veemweg 23 Barneveld 3771 NL paolo@ntt.net

General NMOP YANG-Push Data Mesh Network Telemetry Network Analytics This document specifies a mechanism to define a unique Message key for a YANG to Message Broker integration and a topic addressing scheme based on YANG-Push subscription type and YANG Schema Node Identifier. This enables YANG data consumption of a subset of subscribed YANG data, either per specific YANG data node, identifier or telemetry message type, by indexing and organizing in Message Broker topics. It helps top index the information by using data taxonomy and organizes data in partitions and shards of Message Brokers and time series databases.

Introduction Nowadays network operators are using machine and human readable YANG to model their configurations and monitor YANG operational data from their networks according to . Most network analytic use cases require real-time data and the delivery of near real-time analytical and actionable insights. This imposes high scalability, resilience and low overhead in the data processing pipeline. Accessing the right data for the right use case with minimal overhead and in the shortest period of time is therefore crucial. Network operators organize their data in a Data Mesh according to where a Message Broker, such as Apache Kafka or Apache Pulsar, facilitates the exchange of Messages among data processing components in topics and subjects. Typically, data is being stored in Message Broker topics for several hours or days to facilitate resilience in the data processing chain and addressed in Subjects depending on Schema, enabling a data consumer to address and re-consume previously consumed data again if previously lost. Dimensional data is structured information in a data store. It uses a model of dimension tables to organize business metrics and their descriptive context. This model, developed by Ralph Kimball, simplifies data analysis and reporting by creating denormalized, easy-to-understand structures for quick querying. It is optimized for online analytical processing (OLAP) and data warehouses by using the data taxonomy to scale in partitions and shards. YANG as a data modelling language based on hierarchical tree-based structures facilitates the modelling of dimensional data. This is best shown with YANG Tree Diagrams. An Architecture for YANG-Push to Message Broker Integration specifies an architecture for integrating YANG-Push with Message Brokers for a Data Mesh architecture. describes how the notification messages at a YANG-Push Receiver are being transformed to the Message Broker while specifies to a Message Schema to contextualize telemetry data. However, neither of these documents addresses how these messages should be indexed in a Message Broker, nor define how topics, partitioning and sharding must be used. Due to this missing dimensional indexing for Message Broker stored YANG data, all YANG data is stored in one single Topic. This leads to a round robin distribution across multiple Partitions where each YANG Schema ID is defined as a subject within that topic. Therefore, the entire Topic from all Partitions needs to be consumed first before data selection can be applied. This leads to avoidable data processing overhead which in turn impairs scalability and real-time capabilities, required for certain Network Analytics use cases. YANG telemetry data can be used for several network analytic use cases. Importantly, depending on the use case, only a subset of the subscribed YANG data might be necessary (in time or space). For example, for specific use cases, it is more important to know the current network state, as opposed to have the full series of the state changes over time. In other use cases, instead of consuming data for all network nodes, only a specific network node or network node component requires the YANG monitoring and hence subscription. This document defines how YANG Messages should be indexed and organized in Message Broker topics by leveraging the network node hostname, the YANG datastore name and a YANG Item Identifier for indexing. Then, a YANG-Push subscription type and YANG Schema name for a Message Broker topic naming scheme is defined to better organize YANG data. Network node hostname, YANG datastore name and subtree and xpath filters are part of "ietf-yang-push-telemetry-message" structured YANG data defined in . YANG data nodes are derived based on YANG Schema tree applied subtree and xpath filters and the content of each telemetry message.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology The following terms are used as defined in :

• Network Telemetry
• Network Analytics
• Value
• State
• Change

The following terms are used as defined in :

• Message Broker
• YANG Message Broker Producer
• YANG Message Broker Consumer
• YANG Schema Registry
• YANG Data Consumer

The following terms are used as defined in Apache Kafka and Apache Pulsar Message Broker:

• Subject: Corresponds to a unique schema tree within a Schema Registry and is used to identify Messages within a Topic.
• YANG Schema ID: A unique ID referencing the schema tree for a subject in schema registry. It is used by the YANG Message Broker Producer To serialize and the YANG Message Broker Consumer to deserialize the message.
• Topic: A communication channel for publishing and subscribing messages with one or more subjects and partitions.
• Topic Compaction: The act of compressing messages in a topic to the latest state. As used with Apache Pulsar. Apache Kafka uses the term Log Compaction with identical meaning.
• Partition: Messages in a topic are spread over hash buckets where a hash bucket refers to a partition being stored within one message broker node. Message ordering is guaranteed within a partition.
• Shard: The same as Partition but distributed among multiple message broker nodes. In this document, the term Partition is being used primarily but the described indexing concept equally applies also to Shards.
• Message: A piece of structured data sent between data processing components to facilitate communication in a distributed system
• Message Key: Metadata associated with a message to facilitate deterministic hash bucketing for instantiated YANG data.

The following terms are used as defined in The Log-Structured Merge-Tree scientific paper:

• LSM Tree: Log-Structured Merge-Tree is a data structure with performance characteristics that makes it attractive for providing indexed access to files with high insert volume. LSM trees, like other search trees, maintain key-value pairs.

The following terms are used as defined in Confluent Schema Registry Documentation:

• Schema: A formalized, documented structure that defines the shape and content of the messages exchange.
• Schema ID: A unique identifier of a schema associated to a Message Broker subject.
• Schema Registry: A system where schemas are registered, compared and retrieved.

The following terms are used as defined in :

• Periodic Subscription
• On-change Subscription
• Sync-On-Start
• Xpath Filter
• Subtree Filter

The following terms are used as defined in :

• Notification
• Hostname

The following terms are used as defined in :

• Datastore

The following terms are used as defined in :

• Schema Node Identifier
• Data Node: Such as container, leaf, leaf-list, list, choice and case elements.
• Schema Tree

Solution Design To identify which network node produced which YANG data instance into which Message Broker Topic, Partition and Subject, YANG Message Keys and Indexes are being introduced. These keys enable a deterministic distribution of YANG messages accross Topics and Partitions enabling applications to consume only the needed data from specific topics and partitions. In order to facilitate Message Broker Topic Compaction, a YANG-Push subscription type based topic naming scheme is defined. This segregates statistical (Value), State and State change YANG metrics and facilitates a YANG Message Broker Consumer to use the Topic wild card consumption method to select based on YANG-Push subscription type.

YANG Message Keys and Indexes For topics that carry YANG telemetry messages as defined in , a Message Key MUST be used. If no Message Key is defined then the Messages are distributed in a round robin fashion across partitions. If a Message Key is defined, then the value of the Message Key is being used as input for the Message Broker Producer hash function to distribute across Partitions. Therefore, Message Keys facilitate Message deterministic distribution. The Message Key not only used for Message indexing at the Message Producer but also at the Message Broker for topic compaction. For YANG, the network node hostname, from which YANG datastore the YANG metrics are published from and the YANG data nodes are used to generate the Message Key. The following sections describe how Message Keys are used in both Message producers and Message consumers.

YANG Message Broker Producer YANG data nodes are uniquely identifiable within the YANG Schema tree. defines with "absolute-schema-nodeid" how absolute YANG Schema node identifiers are being crafted locally unique to the YANG module and how YANG data nodes are associated. defines how YANG data nodes can be subscribed with subtree and xpath selection filters. A YANG-Push publisher publishes with "subscription-started" state notifications for each subscription which filter and filter type is being used to the YANG-Push receiver. To generate the Message Key, the "absolute-schema-nodeid" (see ) must be extracted from the YANG-Push subtree or xpath subscription filter in use. If the identifier refers to a YANG list (see ) the list key () is appended to the identifier, separated by a slash. For example, given the XPath filter shown in , the "absolute-schema-nodeid" is "interfaces/interface". Because the interface list has a key named name, the resulting keys for the Message Key are "interfaces/interface/name" plus the YANG data node name of the list which is in this case the name of the interface.

For example, if the following subtree filter is being used, the "absolute-schema-nodeid" is "hardware/component/state". Therefore, the keys used for the Message Key generation are "hardware/component/name/state" plus the YANG data node name of the list which is in this case the name of the component.

]]>

When the Message is being produced to the Message Broker, the Network node hostname and YANG datastore name is used from the structured YANG data defined in "ietf-yang-push-telemetry-message" where the YANG "absolute-schema-nodeid" with the optional list key is derived from subtree and xpath filters, respectively from their YANG Schema tree.

YANG Message Broker Consumer The consumer hashes the Message Key, applies modulo with the number of partitions, and determines the partition from which it should consume messages bearing that Message Key. At a YANG data store, such as a Time Series database or stream processor, the YANG data could than be ingested into tables according to topic names and indexed per Message Key. If Topic Compaction is enabled, only current state is consumed.

Time Series Database Depending if the YANG Data Consumer knows the Message Key from the YANG Message Broker Consumer or the YANG Schema from the YANG Schema Registry the network telemetry messages can be indexed in a Time series database. The Message Key could serve as the primary key, while keys from the YANG data taxonomy can be reflected in the indexing scheme using primary and secondary keys in a time series database. Implementation examples can be found under .

YANG-Push Message Broker Topic Naming YANG data can be subscribed "periodic", on-change" or "on-change" with "sync-on-start". Periodic subscriptions are used for obtaining statistical metrics. On-Change subscriptions are used for obtaining State Changes and on-change with sync-on-start is used for obtaining States. Message Brokers topics are addressed with a unique name. Usually topics are named hierarchically similar to the DNS namespace where "." delimits hierarchies. This document defines "statistics", "states" and "state-changes" in the topic name as the first part to denote the types of data. Followed by "yang" to denote YANG data. Followed by the YANG prefix and absolute-schema-nodeid where all subsequent "/" are substituted by "_". For example, if the "ietf-interface:interfaces/interface" xpath filter is being used, the Message Broker topic name would be as following. In the example the project name and environment (prod, dev, test etc.) is prefixed.

YANG Message Broker Producer For Message Broker topic creation, the "periodic", "on‑change", and "sync‑on‑start" update triggers contain data defined by the “ietf‑subscribed‑notifications” YANG module (). Subscription state notifications MUST be used to derive the subscribed YANG data when it represents "statistics", "states", or "state‑changes". The YANG "absolute-schema-nodeid" MUST be derived from subtree and xpath filter data of subscription state notifications, respectively from their YANG Schema tree.

YANG Message Broker Consumer The consumer can use a wildcard (*) in the topic name to subscribe to multiple topics. For example, if YANG states should be consumed and indexed in Time Series database or stream processor than below Topic Name could be used, and the YANG data could be ingested into tables according to topic names and indexed per Message Key. If Topic Compaction is enabled, only current state is consumed.

Message Broker Implementations Topic, Partitioning and Message Keys are generic concepts of Message Brokers. There are two known Message Broker implementations supporting all features described in this document.

Apache Kafka Apache Kafka supports Message Keys, Partitioning and Log Compaction. With the following example from the Apache Kafka admin client API a new compacted Topic can be created. future = result.values().get(topicName); // Call get() to block until the topic creation is complete or has // failed if creation failed the ExecutionException wraps the // underlying cause. future.get(); } ]]> The most important configuration items from are "topicName" defines the Topic name, "partitions" the amount of partitions, "replicationFactor" how many times the partition is being replicated. With "compact" in "cleanup.policy" the log compaction can be turned on per topic. With "min.cleanable.dirty.ratio" and "delete.retention.ms" how often and when Log Compaction should occur per topic. Where with "retention.bytes" and with "retention.ms" the topic specific compaction configurations can be limited how often the topics are compacted. The topic names are constrained to 249 character length and the following characters: "a-z", "A-Z", "0-9", ".", "_" and "-". Topics can be created on the fly by producing into a new Topic when "auto.create.topics.enable" has been configured prior. Topics should be deleted at the end of the lifecycle through the "kafka-topics.sh" command. The Partition count for a given Topic can be increased but not decreased. Consumer groups are automatically re-joined and partitions are being rebalanced on Message Broker nodes when Partition count changed.

Apache Pulsar Apache Pulsar supports Message Keys, Partitioning and Topic Compaction. With "brokerServiceCompactionThreshold" when Topic Compaction should occur is being configured. The topic names allow all characters except: "/". Topics can be created on the fly by producing into a new Topic when "allowAutoTopicCreation" has been configured prior. Topics should be deleted at the end of the lifecycle through pulsar-admin or pulsarctl tools. The Partition count for a given Topic can be increased but not decreased. Consumer groups are automatically re-joined and partitions are being rebalanced on Message Broker nodes when Partition count changed.

Time Series Database Implementations Tables, partition and keys are generic concepts of time series databases. With ClickHouse, this document provides examples of how YANG message keys can be obtained from the Message Broker and used for indexing.

ClickHouse

Data Model Unlike other realtime analytics databases, ClickHouse does not (necessarily) rely on partitioning data by timestamp. ClickHouse represents data in the MergeTree format, which is similar to a LSM tree: A table consists of data parts sorted by primary key. When data is inserted in a table, separate data parts are created and each of data part is lexicographically sorted by primary key. For example, if the primary key is ("MessageKey", "Date"), the data in the part is sorted by "MessageKey", and within each "MessageKey", it is ordered by "Date". Data belonging to different partitions are separated into different parts. In the background, ClickHouse merges data parts for more efficient storage. Parts belonging to different partitions are not merged. The merge mechanism does not guarantee that all rows with the same primary key will be in the same data part. Each data part is logically divided into granules. A granule is the smallest indivisible data set that ClickHouse reads when selecting data. ClickHouse does not split rows or values, so each granule always contains an integer number of rows. The first row of a granule is marked with the value of the primary key for the row. For each data part, ClickHouse creates an index file that stores the marks. For each column, whether it's in the primary key or not, ClickHouse also stores the same marks. These marks let you find data directly in column files. Thus, it is possible to quickly run queries on one or many ranges of the primary key.

Message Broker Integration ClickHouse integrates with Message Brokers through Integration Table Engines. Reading (selecting) data through Kafka Table Engine follows Apache Kafka semantics of advancing the offset, so subsequent reads will start at the offset the previous read left off. It is the responsibility of the data model designer to transfer data to a regular table:

• Use the engine to create a Kafka consumer and consider it a data stream.

Example:

• Create a table with the desired structure.

Example:

• Create a materialized view that converts data from the engine and puts it into a previously created table.

The Message Key and partition ID are available as virtual (read only) columns _key and _partition.

Message Formats ClickHouse supports numerous Message formats natively. The example above uses the JSON Lines format but other (binary) formats, such as Apache Avro or Protobuf, are supported as well.

Schema Registry ClickHouse has built in Schema Registry support. For Apache Avro, the Schema Registry and authentication are encoded in additional parameters to the Apache Kafka consumer. For formats such as Confluent JSON_SR, use the "kafka_schema_registry_skip_bytes" parameter to skip reading the Schema Registry preamble. The Schema can then be encoded explicitly.

IANA Considerations This document includes no request to IANA.

Security Considerations This document should not affect the security of the Internet.

Operational Considerations The YANG Message Broker Producer of a YANG-Push receiver should have three config knobs facilitate the features described in this document as optional:

• Topic Distribution: Select between "topic" and "subject" distribution. Default is subject to remain backward compatibility to .
• Distribution Type: Select between "none" and "YANG-Push subscription type".
• YANG Message Key: Select between "enable" and "disable".

Subject distribution enables message ordering for a set of YANG Message Keys on each partition. Where in topic distribution messages are randomly being distributed among partitions. To accommodate for potential date loss throughout the data processing pipeline, periodic update of the current State for State metrics is RECOMMENDED. This can be accommodated with YANG-Push as defined in by complementing "on-change sync on start" subscriptions with "periodic" subscriptions. Alternatively, in YANG-Push Lite defined in this simplified in one subscription.

References Normative References Informative References IEEE IEEE O'Reilly Media Wiley Acta Informatica Apache Software Foundation Apache Software Foundation Confluent Community and Apache Software Foundation

Acknowledgements Thanks to Camilo Cardona, Rob Wilton, Holger Keller, Reshad Rahman, Nigel Davis, Olga Havel and Michael Mackey for their comments and reviews. We also like to thank Victor Lopez for the initial idea on the network controller use case. Ashley Woods, Sivakumar Sundaravadivel and Rafael Julio for the idea of grouping topics by YANG-Push subscription type and insisting that Topic Compaction is a key enabler for inventory metrics and YANG data consumer integration and should be supported day 1. Nigel Davis for confirming that Topic Compaction simplifies indeed data processing system architecture and Loïc Monney for the operational configuration and monitoring details on Apache Kafka.

Contributors Many thanks goes to Hellmar Becker who contributed and on how YANG Message Keys can be obtained from Message Broker, how time series databases can use it for indexing YANG data and example implementation in ClickHouse. ClickHouse

601 Marshall Street Redwood City CA 94063 US hellmar.becker@clickhouse.com