Internet DRAFT - draft-thomy-ntv-tab
draft-thomy-ntv-tab
Internet Engineering Task Force P. THOMY
Internet-Draft Loco-labs
Intended status: Informational 19 December 2023
Expires: 21 June 2024
NTV tabular format (NTV-TAB)
draft-thomy-ntv-tab-00
Abstract
This document describes a set of simple rules for unambiguously and
concisely encoding semantic tabular and multidimensional data (NTV-
TAB format). These rules are based on the NTV structure and its JSON
representation (JSON-NTV format).
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 21 June 2024.
Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
THOMY Expires 21 June 2024 [Page 1]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Presentation . . . . . . . . . . . . . . . . . . . . . . 3
1.2. Key design features . . . . . . . . . . . . . . . . . . . 3
1.3. Conventions Used in This Document . . . . . . . . . . . . 4
2. Tabular data . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1. Principles . . . . . . . . . . . . . . . . . . . . . . . 4
2.2. Tabular structure . . . . . . . . . . . . . . . . . . . . 5
2.3. Field structure . . . . . . . . . . . . . . . . . . . . . 8
2.4. Representation . . . . . . . . . . . . . . . . . . . . . 8
3. NTV-TAB format . . . . . . . . . . . . . . . . . . . . . . . 8
3.1. NTV structure . . . . . . . . . . . . . . . . . . . . . . 8
3.2. simple NTVfield formats . . . . . . . . . . . . . . . . . 9
3.3. default NTVfield formats . . . . . . . . . . . . . . . . 9
3.4. Optimized NTVfield formats . . . . . . . . . . . . . . . 12
3.5. Synthesis . . . . . . . . . . . . . . . . . . . . . . . . 13
4. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.1. Field examples . . . . . . . . . . . . . . . . . . . . . 15
4.2. Dataset examples . . . . . . . . . . . . . . . . . . . . 17
5. Properties . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.1. JSON representation . . . . . . . . . . . . . . . . . . . 19
5.2. Dataset size . . . . . . . . . . . . . . . . . . . . . . 19
5.3. Nested NTV-TAB structure . . . . . . . . . . . . . . . . 20
6. Parsing a JSON-value . . . . . . . . . . . . . . . . . . . . 20
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21
8. Security Considerations . . . . . . . . . . . . . . . . . . . 22
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 22
9.1. Normative References . . . . . . . . . . . . . . . . . . 22
9.2. Informative References . . . . . . . . . . . . . . . . . 22
Appendix A. Dataset sizing . . . . . . . . . . . . . . . . . . . 23
A.1. Methodology . . . . . . . . . . . . . . . . . . . . . . . 23
A.2. Formats . . . . . . . . . . . . . . . . . . . . . . . . . 25
Appendix B. Table schema compatibility . . . . . . . . . . . . . 27
B.1. Table schema . . . . . . . . . . . . . . . . . . . . . . 27
B.2. Compatibility . . . . . . . . . . . . . . . . . . . . . . 28
B.3. Example . . . . . . . . . . . . . . . . . . . . . . . . . 29
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 30
Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 30
1. Introduction
THOMY Expires 21 June 2024 [Page 2]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
1.1. Presentation
The main operational standard used to exchange textual tabular data
is CSV format [RFC4180]. Unfortunately CSV format is obsolete (last
revision in 2005) and current CSV tools do not comply with the
standard.
It is therefore important to define an alternative format that meets
the expectations of tabular and multidimensional data exchanges. The
NTV-TAB format proposed here is a response to this need.
1.2. Key design features
The format's focus is on simplicity, lightness and web usage.
The key features of this format are the following:
* JSON as the base format
JSON is simple and readable as simple text
JSON supports rich structure including nesting and basic types
JSON is web-native and very widely used and supported
JSON format has binary representation (i.e. CBOR format)
* optimized representations
from the simplest to the most optimized, are available
avoid data duplication,
reduce the size of data,
allows strict and unambiguous reversibility (lossless round-
trip)
* high semantic level of data (JSON-NTV as a grammar)
Take into account most common data formats used in Internet
standards
wide variety of data typing
meta-data (header or schema) can be integrate
common format between tabular and multidimensional data
THOMY Expires 21 June 2024 [Page 3]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
* simple, compact, extensible and self-describing
1.3. Conventions Used in This Document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
This document also uses the following terms:
*JsonText, JsonValue, JsonObject, JsonMember, JsonElement,
JsonArray, JsonNumber, JsonString, JsonFalse, JsonNull, JsonTrue
:*
These terms are defined in [JSON-NTV].
*NTV, NTVlist, NVlist, Vlist, TVlist, NTVsingle, NVsingle,
TVsingle, Vsingle, NTVname, NTVtype, NTVvalue, JsonNTVtype,
JsonNTVname, JsonPrimitive, JsonUnnamed, JsonNamed:*
These terms are defined in [JSON-NTV].
*Row, Column, Table, Cell:*
These terms are defined in [W3C_TAB].
*Dataset*
A Dataset is equivalent to a Table
*Field*
A Field is equivalent to a Column
2. Tabular data
2.1. Principles
_*Tabular data* is data that is structured into rows, each of which
contains information about some things. Each row contains the same
number of cells (although some of these cells may be empty), which
provide values of properties of the thing described by the row. In
tabular data, cells within the same column provide values for the
same property of the things described by each row. This is what
differentiates tabular data from other line-oriented
formats._[W3C_TAB]
Two main uses are identified for tabular data:
THOMY Expires 21 June 2024 [Page 4]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
* a flow-oriented use for which each row is independent of the
others. The dataset is then seen as a list of rows whose number
can be variable. This is for example the case of a list of
measurements from a sensor.
* a structure-oriented use for which the rows are not independent
and contribute to describing the same object. This is for example
the case of a grade table for a class which integrates the
students, courses, periods, etc.
This document deals with this second use.
2.2. Tabular structure
In structure-oriented use, columns and rows are not equivalent, the
columns (or Fields) represent the 'semantics' of the data and the
rows represent a specific combination of Field's values according to
the structure defined by the tabular data (Dataset). The nature of
the rows is often implicit.
Two basic patterns are present in Datasets:
* *Tree pattern*: A tree is represented in tabular form by a list of
paths between each leaf and the node. The columns then represent
the levels of the tree.
* *Matrix pattern*: A matrix (or multidimensional data) is
represented in tabular form by a column of the values of the
matrix and additional columns represent the coordinates of each of
the values.
Table 1 and Table 2 present an example of such patterns
+======+=========+=========+
| Root | level 1 | level 2 |
+======+=========+=========+
| A | B | D |
+------+---------+---------+
| A | B | E |
+------+---------+---------+
| A | C | F |
+------+---------+---------+
| A | C | G |
+------+---------+---------+
Table 1: Tree pattern
THOMY Expires 21 June 2024 [Page 5]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
+=======+=====+=====+
| Value | row | col |
+=======+=====+=====+
| 1 | A | C |
+-------+-----+-----+
| 2 | A | D |
+-------+-----+-----+
| 3 | B | C |
+-------+-----+-----+
| 4 | B | D |
+-------+-----+-----+
Table 2: Matrix pattern
Taking these structures into account leads to significant duplication
of data. In the general case, Datasets mix these different
structures.
If we now observe the relationships between Fields [TAB-ANA], we can
identify four main uses:
* *association*: this consists of coupling each value of a Field to
a single value of another Field ("coupled" relationship between
two fields),
* *classification*: This involves grouping the data by category in
order - for example - to be able to make a statistical use of it,
("derived" relationship between two fields),
* *crossing*: This consists of representing all the combinations
between the two Fields, such as in matrix representations
("crossed" relationship between two fields),
* *characterization*: It corresponds to the documentation of defined
properties (no specific relationship).
_Example: Price list of different foods based on packaging for the
year 2022._Table 3
THOMY Expires 21 June 2024 [Page 6]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
+==+=======+=========+=========+======+=====+========+==============+
|Id|Product|Food |Packaging|Weight|Price| Period | Availability |
+==+=======+=========+=========+======+=====+========+==============+
|11|apple |fruit |bag |1 kg |1 | 2nd | Yes |
| | | | | | | half | |
| | | | | | | 2022 | |
+--+-------+---------+---------+------+-----+--------+--------------+
|12|apple |fruit |cardboard|10 kg |9 | 2nd | Yes |
| | | | | | | half | |
| | | | | | | 2022 | |
+--+-------+---------+---------+------+-----+--------+--------------+
|13|orange |fruit |bag |1 kg |2 | 2nd | end of 2022 |
| | | | | | | half | |
| | | | | | | 2022 | |
+--+-------+---------+---------+------+-----+--------+--------------+
|14|orange |fruit |cardboard|10 kg |18 | 2nd | end of 2022 |
| | | | | | | half | |
| | | | | | | 2022 | |
+--+-------+---------+---------+------+-----+--------+--------------+
|15|pepper |vegetable|bag |1 kg |1.5 | 2nd | end of 2022 |
| | | | | | | half | |
| | | | | | | 2022 | |
+--+-------+---------+---------+------+-----+--------+--------------+
|16|pepper |vegetable|cardboard|10 kg |13 | 2nd | end of 2022 |
| | | | | | | half | |
| | | | | | | 2022 | |
+--+-------+---------+---------+------+-----+--------+--------------+
|17|banana |fruit |bag |1 kg |0.5 | 2nd | Yes |
| | | | | | | half | |
| | | | | | | 2022 | |
+--+-------+---------+---------+------+-----+--------+--------------+
|18|banana |fruit |cardboard|10 kg |4 | 2nd | Yes |
| | | | | | | half | |
| | | | | | | 2022 | |
+--+-------+---------+---------+------+-----+--------+--------------+
Table 3: Price list
_We find here:_
* _association: between "Packaging" and "Weight",_
* _classification: between "Product" and "Food",_
* _crossing: between "Product" and "Weight",_
* _characterization: between "Product" and "Availability"_
THOMY Expires 21 June 2024 [Page 7]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
2.3. Field structure
A Field is an ordered set of Cells.
To represent this structure, several representations are possible
depending on the nature of the data:
* the simplest format is to represent a Field by the list of Cells
with the same order for all Fields. This format is interesting
when the data is little duplicated,
* when the data is repetitive, a second option is to represent on
the one hand the list of different data and on the other hand
their position in the list (i.e. categorical data),
* another special case also concerns repetitive data for which one
value is highly predominant (sparse data). In this case, it is
sufficient to provide only the position in the list of data except
for the one that is predominant,
* a last option consists in representing the Field according to its
dependence with another Field (coupled, derived or crossed
relationship). This leads to an optimized data volume.
2.4. Representation
Three representations are available for a tabular object : row-
oriented (list of Rows), cells-oriented (list of Cells), field-
oriented (list of Fields).
The field-oriented representation is retained because it takes into
account the semantics carried by the Fields as well as the inter-
Field analysis presented above.
A Dataset is then seen as a set of Fields representing the properties
of the entire Dataset.
The order of Fields or Rows is not relevant.
3. NTV-TAB format
3.1. NTV structure
A Dataset is represented by the following NTV entities:
* NTVcell represents a Cell. NTVcell is a NTVsingle.
THOMY Expires 21 June 2024 [Page 8]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
* NTVfield represents a Field. NTVfield is a NTV entity depending
on the format chosen to represent the Field (simple format,
default format, optimized format). A NTVfield contains a NTVlist
of part of the NTVcells (Codec) and optionnaly coding data.
* NTVdataset represents the Dataset. NTVdataset is a NVlist where
the NTVname is the name of the Dataset and the NTVvalue is the
list of NTVfields.
The JSON format of a NTVdataset is his JSON-NTV format.
3.2. simple NTVfield formats
This category is the usual representation of a Field with different
values (Full format) or with several identical values (Unique
format).
*Full format* :
The Full format is the format that does not use any coding. Codec
and NTVfield are identical. The NTVfield is therefore a NTVlist
where the NTVname is the name of the Field, the NTVtype is the
default type of the NTVcells and the NTVvalue is the list of
NTVcells.
_Example JsonNTVvalue ( "price" Field)_ :
_[ 1, 9, 2, 18, 1.5, 13, 0.5, 4 ]_
*Unique format* :
The Unique format is used when all NTVcells are identical. The Codec
is the NTVcell.Codec and NTVfield are identical (coding is implict).
The NTVfield is therefore the NTVcell.
_Example JsonNTVvalue ( "period" Field)_ :
_"2nd half 2022"_
Note :
The Unique format also makes it possible to represent tabular
metadata
3.3. default NTVfield formats
This category completes the simple formats with the other most common
representations of a Field :
THOMY Expires 21 June 2024 [Page 9]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
* Categorical Field (Complete format)
* Periodic Field (Primary format)
* Sparse Field (Sparse format)
In those formats, Codec is explicit and is the TVlist of different
Field NTVcells (Codec). The NTVfield is a NVlist where the NTVname
is the name of the Field.
*Complete format* :
The "complete format" is equivalent to the format used to store
categorical variables.
The NTVfield is a NVlist composed with two NTV entities :
* Codec: TVlist of different Field NTVcells (Codec),
* Coding: Vlist of indexes of NTVcells in Component (Keys)
The list of NTVcells is reconstituted by replacing the integers in
the coding Vlist with the NTVcell at the coding index in the Codec
(e.g. pandas categories and codes).
_Example JsonNTVvalue ( "product" Field)_ :
_[ [ "orange" , "pepper" , "apple" , "banana" ], [ 2, 2, 0, 0,
1, 1, 3, 3 ] ]_
*Sparse format* :
A specific format (one dimensional sparse LIL format) is used for
sparse data. It is defined by:
* 'fill_value': it should be most common value
* 'sp_value': it is a list storing only values distinct from the
'fill_value'
* 'sp_index': list of index of 'sp-value' in the sparse data list
The NTVfield is a NVlist composed with three NTV entities :
* Codec: TVlist of different Field NTVcells (Codec),
* Ref: Vlist of indexes of Codec value in 'sp_value' ,
THOMY Expires 21 June 2024 [Page 10]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
* Coding: Vlist of 'sp_index'
The list of NTVcells is reconstituted by replacing in a list of
'fill_value', the values with index in the Coding Vlist by the
corresponding value defined by the Ref index in the Codec TVlist.
_Example JsonNTVvalue ( "food" Field)_ :
_[ [ "vegetable" , "fruit"], [0, 0], [ 4, 5 ] ] 'fruit' is the
'fill_value' - 0 is the index of "vegetable"_
*Primary format* :
This format is equivalent to the Complete format where the Keys Vlist
is calculated with the "repetition coefficient".
The NTVfield is a NVlist composed with two NTV entities :
* Codec: TVlist of different Field NTVcells (Codec),
* Coding: Vlist with a single integer (Repetition coefficient)
The Keys Vlist is generated with the formula:
keys[ikey] = ( ikey % ( coef * period ) ) // coef
where:
keys: is the Keys Vlist
ikey: is the index of a key value
coef: is the Repetition coefficient
period: is the length of Codec
_Example: coef = 2, period = 3, Keys length = 12_
_keys = [0, 0, 1, 1, 2, 2, 0, 0, 1, 1, 2, 2]_
The Repetition coefficient is the number of adjacent identical values
in the Keys list.
_Example "packaging"_ :
_Codec: [ "bag" , "cardboard" ]_
_Coefficient: 1_
THOMY Expires 21 June 2024 [Page 11]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
_(implicit Keys : [ 0, 1, 0, 1, 0, 1, 0, 1 ] )_
_Example "product"_ :
_Codec: [ "apple" , "orange" , "pepper" , "banana" ]_
_Coefficient: 2_
_(implicit Keys : [ 0, 0, 1, 1, 2, 2, 3, 3 ] )_
3.4. Optimized NTVfield formats
This category of formats reduces the size of Complete format with
optimized Keys. The length of Keys is reduced with using of derived
(Relative format) or coupled (Implicit format) relationships between
two Fields.
In those formats, Codec is explicit and is the TVlist of different
Field NTVcells (Codec). The NTVfield is a NVlist where the NTVname
is the name of the Field.
*Implicit format* :
This representation is associated with "coupled" Fields. These
Fields have a one-to-one correspondence.
The NTVfield is a NVlist composed with two NTV entities :
* Codec: TVlist of different field NTVcells (Codec),
* Ref: Vsingle entity index or name of the coupled Field.
This format is equivalent to the Complete format where Keys is the
Keys of the Field (with Complete format) defined by Ref.
_Example JsonNTVvalue ( "weight" Field is associated with
"packaging" Field )_ :
_[ [ "1 kg" , "10 kg" ], "packaging"]_
_( implicit Keys : [ 0, 1, 0, 1, 0, 1, 0, 1 ] )_
*Relative format* :
This representation is associated with "derived" Fields. These
Fields have a one-to-many correspondence.
THOMY Expires 21 June 2024 [Page 12]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
The values of a "derived" Field are inferred from the values of the
parent Field.
The Field is a NVlist composed with three NTV entities :
* Codec: TVlist of different field NTVcells (Codec),
* Ref: Vsingle entity index or name of the parent Field,
* Coding : Vlist of relative indexes of NTVcells in Codec (Relative
Keys).
This format is equivalent to the Complete format where the Keys Vlist
is obtained by replacing the values of the Keys Vlist of the parent
Field with the corresponding values in the Relative Keys (the length
of the Relative Keys is the length of the Codec of the parent Field).
_Example JsonNTVvalue ( "food" Field - "product" Field is the
parent Field of "food" Field)_ :
_[ [ "fruit" , "vegetable" ], "product", [ 0, 1, 0, 0 ] ]_
_(the Vlist Keys is obtained by replacing the values 0, 1, 2, 3
of the Vlist Keys of the "product" Field by the values 0, 1, 0,
0 of the Relative Keys i.e.: [ 0, 0, 0, 0, 1 , 1, 0, 0] )_
3.5. Synthesis
The NTVfield structure corresponding to the format defined above are
in Table 4:
THOMY Expires 21 June 2024 [Page 13]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
+====================+========+===============+==================+
| Structure | Codec | Ref | Coding |
+==========+=========+========+===============+==================+
| format | NTV | TVlist | Vsingle | Vlist |
+==========+=========+========+===============+==================+
| Relative | NTVlist | x | index | Relative Keys |
| | | | | |
| | len = 3 | | or name | len < len(Field) |
+----------+---------+--------+---------------+------------------+
| Complete | NTVlist | x | | Keys |
| | | | | |
| | len = 2 | | | len = len(Field) |
+----------+---------+--------+---------------+------------------+
| Sparse | NTVlist | x | list of index | sp_index |
| | | | | |
| | len = 3 | | sp_value | 1<len<len(Field) |
+----------+---------+--------+---------------+------------------+
| Implicit | NTVlist | x | index | |
| | | | | |
| | len = 2 | | or name | |
+----------+---------+--------+---------------+------------------+
| Primary | NTVlist | x | | coef |
| | | | | |
| | len = 2 | | | len = 1 |
+----------+---------+--------+---------------+------------------+
| Unique | NTVsingle |
+----------+-----------------------------------------------------+
| Full | NTVlist len = len(Field) |
+----------+-----------------------------------------------------+
Table 4: NTVfield formats
Three levels are available to convert tabular data in JSON structure
Table 5.
* *Level 0: "simple"* is the usual representation of tabular data.
Fields are converted with the Simple or Unique format.
* *Level 1: "default"* avoids duplication of information by adding
simple encoding.
Fields are converted according to their own structure (simple,
unique, categorical, sparse, periodic).
* *Level 2: "optimize"* avoids duplication of information and
minimizes encoding. It is the usual representation of
multidimensional data.
THOMY Expires 21 June 2024 [Page 14]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
This level requires an analysis of the relationships between
Fields ("partition")
+===============+=========================+
| Level | Structure |
+====+==========+==============+==========+
| | mode | Type Field | format |
+====+==========+==============+==========+
| 0 | simple | Unique | Unique |
| | +--------------+----------+
| | | Simple | Full |
+----+----------+--------------+----------+
| 1 | default | Unique | Unique |
| | +--------------+----------+
| | | Simple | Full |
| | +--------------+----------+
| | | Sparse | Sparse |
| | +--------------+----------+
| | | Categorical | Complete |
| | +--------------+----------+
| | | Periodic | Primary |
+----+----------+--------------+----------+
| 2 | optimize | Unique | Unique |
| | +--------------+----------+
| | | Root coupled | Full |
| | +--------------+----------+
| | | Root derived | Complete |
| | +--------------+----------+
| | | Primary | Primary |
| | +--------------+----------+
| | | Derived | Relative |
| | +--------------+----------+
| | | Coupled | Implicit |
+----+----------+--------------+----------+
Table 5: NTVfield levels
4. Examples
4.1. Field examples
The example in Section 2.2 has the following JSON representation
Table 6:
THOMY Expires 21 June 2024 [Page 15]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
+========+========================================================+
| Format | JsonNTV Representations |
+========+========================================================+
|Full |{ "price::float": [ 1, 9, 2, 18, 1.5, 13, 0.5, 4 ] } |
| | |
| |{ "price": [ 1, 9, 2, 18, 1.5, 13, 0.5, 4 ] } |
| | |
| |[ 1, 9, 2, 18, 1.5, 13, 0.5, 4 ] |
+--------+--------------------------------------------------------+
|Complete|{"product":[["orange","pepper","apple","banana"], |
| | |
| |[2,2,0,0,1,1,3,3]]} |
| | |
| |{"product": [ ["orange","pepper","apple","banana"], |
| | |
| |[2, 2, 0, 0, 1, 1, 3, 3] ]} |
| | |
| |[ ["orange","pepper","apple","banana"], |
| | |
| |[2, 2, 0, 0, 1, 1, 3, 3] ] |
+--------+--------------------------------------------------------+
|Unique |{ "period": "2nd half 2022" } |
| | |
| |"2nd half 2022" |
+--------+--------------------------------------------------------+
|Implicit|{"weight":[{"::string":["1 kg","10 kg"]},"packaging"]} |
| | |
| |[["1 kg","10 kg"],3] |
+--------+--------------------------------------------------------+
|Relative|{"food": [ {"::string": [ "fruit" , "vegetable" ]}, |
| | |
| |"product", [ 0, 1, 0, 0 ]] } |
| | |
| |[ [ "fruit" , "vegetable" ], 1, [ 0, 1, 0, 0 ] ] |
+--------+--------------------------------------------------------+
|Sparse |{"food":[{"::string":["vegetable","vegetable","fruit"]},|
| | |
| |[4,5,-1]]} |
| | |
| |[["vegetable","vegetable","fruit"],[4,5, 1]] |
+--------+--------------------------------------------------------+
|Primary |{"packaging":[{"::string":["cardboard","bag"]},[1]]} |
| | |
| |[["cardboard","bag"],[1]] |
| | |
| |{"product":[["apple","orange","peppers","banana"],[2]]} |
| | |
| |[["apple","orange","peppers","banana"],[2]] |
THOMY Expires 21 June 2024 [Page 16]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
+--------+--------------------------------------------------------+
Table 6: NTVfield examples
4.2. Dataset examples
The examples in Table 7 below illustrate the optimize level:
THOMY Expires 21 June 2024 [Page 17]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
+====================================+==============================+
| Data | Optimize level |
+========+===========================+==============================+
| type | Full format | JsonNTV |
+========+===========================+==============================+
|matrix |[['a','a','b','b','c','c'],|[[['a','b','c'],[2]], |
| | | |
| |[10,20,10,20,10,20], |[[10,20],[1]], |
| | | |
| |[1,2,3,4,5,6]] |[1,2,3,4,5,6]] |
+--------+---------------------------+------------------------------+
|single |[[1,2,3,4,5,6], |[[1,2,3,4,5,6], |
| | | |
| |['a','a','a','a','a','a']] |'a'] |
+--------+---------------------------+------------------------------+
|complete|[[1,2,3,3,5,5]] |[[[1,2,3,5],[0,1,2,2,3,3]]] |
+--------+---------------------------+------------------------------+
|coupled |[[1,2,3,3,5,5], |[[[1,2,3,5],[0,1,2,2,3,3]], |
| | | |
| |['a','b','c','c','e','e']] |[['a','b','c','e'],0]] |
+--------+---------------------------+------------------------------+
|derived |[[1,2,3,4,5,6], |[[1,2,3,4,5,6], |
| | | |
| |['a','a','b','b','c','c'], |[['a','b','c'],[0,0,1,1,2,2]],|
| | | |
| |[10,10,10,10,20,20]] |[[10,20],1,[0,0,1]]] |
+--------+---------------------------+------------------------------+
|matrix |[[6,6,7,7,8,8,9,9], |[[[6,7,8,9],[2]], |
| | | |
|+ |[10,20,10,20,10,20,10,20], |[[10,20],[1]], |
| | | |
|coupled |[1,1,2,2,3,3,4,4], |[[1,2,3,4],0], |
| | | |
| |[1,2,3,4,5,6,7,8]] |[1,2,3,4,5,6,7,8]] |
+--------+---------------------------+------------------------------+
|matrix |[[6,6,7,7,8,8,9,9], |[[[6,7,8,9],[2]], |
| | | |
|+ |[10,20,10,20,10,20,10,20], |[[10,20],[1]], |
| | | |
|coupled |[1,1,2,2,3,3,4,4], |[[1,2,3,4],0], |
| | | |
|+ |[11,11,22,22,22,22,22,22], |[[11,22],0,[0,1,1,1]], |
| | | |
|derived |[1,2,3,4,5,6,7,8]] |[1 2,3,4,5,6,7,8]] |
+--------+---------------------------+------------------------------+
Table 7: optimize level examples
THOMY Expires 21 June 2024 [Page 18]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
The examples in Table 8 below illustre NTVdataset with a length equal
to 0, 1 or 2:
+------------------+-------------------------------------------+
| [ ] or { } | _Empty NTVdataset_ |
+------------------+-------------------------------------------+
| [25] or [[25]] | _NTVdataset with 1 NTVfield and length 1_ |
+------------------+-------------------------------------------+
| [2, 1] or [[2], | _NTVdataset with 2 NTVfield and length 1_ |
| [1]] or [2, [1]] | |
+------------------+-------------------------------------------+
| [[2, 1]] | _NTVdataset with 1 NTVfield and length 2_ |
+------------------+-------------------------------------------+
| [[2, 1], [4, 3]] | _NTVdataset with 2 NTVfield and length 2_ |
+------------------+-------------------------------------------+
Table 8: NTVdataset with length 0, 1 or 2
5. Properties
5.1. JSON representation
NTV-TAB format defines the representation of a Dataset into the NTV
format. This conversion is reversible (lossless).
Furthermore, the NTV format defines the conversion into JSON format.
This conversion is also reversible.
The exchange format (JsonText) of a Dataset is therefore obtained by
a representation in NTV-TAB format then a conversion to JSON format
and finally a conversion to text format (or binary format with CBOR
conversion). The data is reconstituted identically by reverse
conversions.
5.2. Dataset size
As explain in Section 2.2 cells are often duplicated in a Field. The
principle of NTV-TAB format is to replace duplicated data with
encoding based on integers.
This optimization considerably reduces the size of a representation
of a Dataset. Appendix A details the methodology to optimize this
size.
THOMY Expires 21 June 2024 [Page 19]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
5.3. Nested NTV-TAB structure
NTVcells in a NTVdataset are any NTVsingle. We can therefore include
in a NTVdataset the data associated with the types defined in the NTV
format.
The 'tab' and the 'field' NTVtypes are associated to NTVdataset and
NTVfield. A NTVcell can also include a NTVdataset or a NTVfield.
Figure 1 is an example of nested Dataset. The 'nested' JsonNTV is
the representation of a Dataset with length equal 2 and composed with
two Fields 'field1' and 'field2'.
* 'field1' is a Field with two Cells 'dataset1' ans 'dataset2' which
are Dataset. 'field1' is represented with a Full format NTVfield.
* 'field2' is a Field with two Cells 'field2_1' ans 'field2_2' which
are Field. 'field2' is represented with a Full format NTVfield.
nested = {
"field1": {
"dataset1:tab":{
"dts1_field1": [1,2,3],
"dts1_field2": [4,5,6]
},
"dataset2:tab":{
"dts2_field1": [10,20,30],
"dts2_field2": [40,50,60],
"dts2_field3": [70,80,90]
},
},
"field2":{
"field2_1:field": [1,2,3],
"field2_2:field": [4,5,6],
}
}
Figure 1: Nested Dataset
6. Parsing a JSON-value
A NTV parser generates a NTV entity from a JSON-value.
The decoding NTV entity is directly converted into the NTVdataset and
a list of NTVfields.
For each NTVfield the format is deduced following the structure
defined in the table xxx.
THOMY Expires 21 June 2024 [Page 20]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
For each format, a decoder converts the NTVvalue of the NTVfield into
the chosen object.
_Note :_
_Several NTVvalue are ambiguous to deduce the Field format :_
- _[ list-data, integer, list-integer ] : Full or Relative format
?_
- _[ list-data, string, list-integer ] : Full or Relative format
?_
- _[ list-data, integer ] : Full or Implicit format ?_
- _[ list-data, string ] : Full or Implicit format ?_
- _[ list-data, list-integer ] : Full or Complete/Primary format
?_
- _[ list-data, list-integer, list-integer ] : Full or Sparse
format ?_
_The full format is not retained for those NTVvalue._
_To avoid this ambiguity, precautions can be taken for Dataset
with length = 2 or 3 and with a Full format:_
- _a name can be added to the list-data (e.g. { "data": list-
data}),_
- _the order of data can be changed (e.g. [ integer, list-data
])_
- _a type can be added (e.g. { "::json": [ list-data, list-
integer ] } )_
- _an additional field can be added (e.g. [ list-data, integer,
list-integer, {"format": "full"} ] )._
7. IANA Considerations
Any JsonValue is a JsonNTVValue and conversely, any JsonNTVvalue is a
JsonValue.
Thus, any JSON data may or may not be treated as JsonNTV data, so
there is no need to create a specific MIME media type for JsonNTV.
THOMY Expires 21 June 2024 [Page 21]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
All properties of the MIME media type "application/json" are
applicable.
8. Security Considerations
The format used for NTV data exchanges is the JSON format. So, all
the security considerations of [RFC8259] apply.
The NTV structure provides no cryptographic integrity protection of
any kind.
9. References
9.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC4180] Shafranovich, Y., "Common Format and MIME Type for Comma-
Separated Values (CSV) Files", RFC 4180,
DOI 10.17487/RFC4180, October 2005,
<https://www.rfc-editor.org/info/rfc4180>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
Interchange Format", STD 90, RFC 8259,
DOI 10.17487/RFC8259, December 2017,
<https://www.rfc-editor.org/info/rfc8259>.
9.2. Informative References
[TABLE] "FrictionLess", "Table Schema", 2021,
<https://specs.frictionlessdata.io/table-
schema/#language>.
[JSON-NTV] Thomy, P., "JSON semantic format (JSON-NTV)", 2023,
<https://datatracker.ietf.org/doc/draft-thomy-json-ntv/>.
[TAB-ANA] Thomy, P., "Tabular dataset analysis", 2022,
<https://github.com/loco-philippe/tab-
analysis/blob/main/docs/tabular_analysis.pdf>.
THOMY Expires 21 June 2024 [Page 22]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
[W3C_TAB] "W3C", "Recommendation : Model for Tabular Data and
Metadata on the Web", 17 December 2015,
<https://www.w3.org/TR/2015/REC-tabular-data-model-
20151217/>.
Appendix A. Dataset sizing
This appendix presents an analysis of NTVdataset size optimization
with the defined formats.
A.1. Methodology
The principle of defined formats is to replace duplicated data with
encoding based on integers.
We define the size of a Dataset representation (SZ) as the sum of the
encoding size and the size of unencoded unduplicated values. The
coding is modeled as being the product of the values remaining to be
represented (nv - nc) with an average coding size (sc):
SZ = nc * sv + (nv - nc) * sc
where :
nv : number of values
sv : mean value size
nc : number of different values
sc : mean coding size
example :
Full format : {"product":
["orange","apple","apple","apple","orange","orange"]}
Complete format : {"product": [ ["orange","apple"], [1, 0, 0,
0, 1, 1] ]}
SZ = 9 + 8 + 7 + 6 * 1 = 30 (including double quotes),
nv = 9 (including the Field name),
sv = (9 + 8 * 3 + 7 * 3) / 7 = 7.71
nc = 3 (including the Field name)
THOMY Expires 21 June 2024 [Page 23]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
sc = (30 - 3 * 7.71) / (9 - 3) = 1.15 (sc = (SZ - nc * sv) /
(nv - nc))
In this example the JSON overhead (coma, space, curly bracket,
square bracket) is not included.
SZ is maximal when there is no coding (sc = sv) and minimal when the
coding is perfect (sc = 0):
SZmax = nv * sv
SZmin = nc * sv
We then define the following indicators:
* unicity level UL = nc / nv
UL = SZmin / SZmax
1 - UL = (SZmax - SZmin) / SZmax
UL characterizes the nature of the data independently of the
coding and represents the maximum achievable gain (1-UL).
maximum UL = 1 (unduplicated data)
minimum UL = 0 (full duplicated data = empty data)
* object lightness OL = sc / sv
OL = (SZ - SZmin) / (SZmax - SZmin)
1 - OL = (SZmax - SZ) / (SZmax - SZmin)
OL characterizes coding efficiency
maximum OL = 1 (no coding)
minimum OL = 0 (perfect coding)
The optimization of the size of the representation is then evaluated
by comparing the size obtained without coding and that obtained with
coding:
G = (SZmax - SZ) / SZmax = (1 - UL) * (1 - OL)
R = 1 - G = SZ / SZmax = UL + OL - UL * OL
THOMY Expires 21 June 2024 [Page 24]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
The maximum G gain is 1 - UL, the minimum G gain is 0.
If the data is empty, UL = OL = 0 and the gain is equal to 1.
The indicators are deduced from the following four measurable values:
* number of cells in the dataset (nv)
* number of different cells in the dataset (nc)
* size of the dataset with the format to study (SZ)
* size of the dataset with Full format (SZmax)
We then deduce sv = SZmax / nv as well as sc = (SZ - nc * sv) / (nv -
nc)
In the example above, the indicators are:
UL = 3 / 9 = 0.33
OL = 1.15 / 7.71 = 0.15
G = 0.67 * 0.85 = 0.57
SZmax = 9 * 7.71 = 69.4
SZmin = 3 * 7.71 = 23.1
The Complete format is close to the minimum size (SZ = 30) and its
size is less than half the size of the Full format (43 %).
A.2. Formats
The formats used to represent an NTVfield are in general form:
* list of part of NTVcells
* list of integers used to encode other NTVcells
The size of this format can then be written (without taking into
account the overhead linked to the format):
SZ = nc * sv + k * nv * si
where :
nv : number of values
THOMY Expires 21 June 2024 [Page 25]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
sv : mean value size
nc : number of different Field values
si : integer size
k: specific coefficient of the coding used
Comparison with the structure defined in the previous chapter allows
us to deduce the parameters:
UL = nc/nv
sc = si * k * nv/(nv-nc)
OL = si/sv * k * nv/(nv-nc)
G = 1- nc/nv - si/sv * k
R = nc/nv + si/sv * k
The gain G is therefore equal to the maximum gain 1-UL reduced by the
weight of the coding corresponding to the parameter k weighted by the
average size of the values compared to an integer.
Table 9 below specifies the values of k for the different formats:
+==========+===============+===========================+
| Format | k coefficient | comments |
+==========+===============+===========================+
| Full | 0 | R = 1 |
+----------+---------------+---------------------------+
| Unique | 0 | R = 1/nv (nc = 1) |
+----------+---------------+---------------------------+
| Complete | 1 | R = nc/nv + si/sv |
+----------+---------------+---------------------------+
| Primary | 1 / nv | R = nc/nv + si/sv/nv |
+----------+---------------+---------------------------+
| Coupled | 1 / nv | R = nc/nv + si/sv/nv |
+----------+---------------+---------------------------+
| Sparse | 2 * ns / nv | R = nc/nv + 2*si/sv*ns/nv |
+----------+---------------+---------------------------+
| Derived | nd / nv | R = nc/nv + si/sv*nd/nv |
+----------+---------------+---------------------------+
Table 9: coding coefficient
_ns: number of values distinct from the 'fill_value'_
THOMY Expires 21 June 2024 [Page 26]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
_nd: number of different values in the parent Field_
Appendix B. Table schema compatibility
This appendix presents the compatibility between Tableschema [TABLE]
and the NTV-TAB format.
B.1. Table schema
Table Schema is a simple language- and implementation-agnostic way to
declare a schema for tabular data. A Table Schema is represented by
a descriptor. The descriptor MUST be a JSON object with defined
properties (JsonMember).
Table Schema define following descriptors and properties:
* Fields property
Fields property MUST be an array where each entry in the array
is a field descriptor (as defined below).
* Field descriptor
A field descriptor MUST be a JSON object that describes a
single field.
* Field properties
The field descriptor object MAY contain any number of other
properties. Some specific properties are defined below. Of
these, only the name property is REQUIRED.
Defined Properties:
o name
o title
o description
o example
o type / format
o constraints
The constraints property on Table Schema Fields can be used by
consumers to list constraints for validating field values.
THOMY Expires 21 June 2024 [Page 27]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
* Table properties
In additional to field descriptors, there are the following
"table level" properties:
o missingValues
o primaryKey
o foreignKeys
B.2. Compatibility
Three levels of compatibility are addressed :
* Concepts
The concepts are equivalent between Table Schema and NTV-TAB :
o Table is equivalent to Dataset
o Field in Table Schema is equivalent to the NTVfield
o Name in Table Schema is equivalent to the NTVname of the
NTVfield
o Type / Format in Table Schema is equivalent to the NTVtype
of the NTVfield
* Type / Format
NTVtype combines the concepts of type and format. The
correspondence table in the NTV specification [JSON-NTV] gives
the link between an NTVtype and the corresponding type/format.
* Constraints
Constraints are applicable to each value in a Table Field.
Validating constraints for all values in a Table Field is
equivalent to validating a constraint for all values in the
Codec list.
These compatibility levels are reached, which makes it possible to
validate an NTVdataset with a schema defined according to the
Table Schema format.
The following principles should then be considered to validate an
NTVdataset:
THOMY Expires 21 June 2024 [Page 28]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
* The NTVfields names must be identical to the Schema names,
* If the NTVtype of Codec data is not 'json', it must match the
Type/Format defined in the Schema,
* If the constrainst are valid with the Codec data, they are valid
with the Field.
B.3. Example
Figure 2 is an example of Dataset with Full format ('tab_data1'
without NTVtypes) and with other formats ('tab_data2').
tab_data1 = {
"index": [100, 200, 300, 400, 500, 600],
"dates": ["1964-01-01", "1985-02-05", "2022-01-21", "1964-01-01",
"1985-02-05", "2022-01-21"],
"value": [10, 10, 20, 20, 30, 30],
"coord": [[1,2], [3,4], [5,6], [7,8], [3,4], [5,6]],
"names": ["john", "eric", "judith", "mila", "hector", "maria"],
"unique": ["true, "true", "true", "true", "true", "true"]
}
tab_data2 = {
"index": [100, 200, 300, 400, 500, 600],
"dates": {"::date":[["1964-01-01","1985-02-05","2022-01-21"],[1]},
"value": [[10, 20, 30], [2]],
"coord::point": [[1,2], [3,4], [5,6], [7,8], [3,4], [5,6]],
"names::string":["john", "eric", "judith", "mila", "hector",
"maria"],
"unique": True
}
Figure 2: Dataset example
The schema in Figure 3 is valid with 'tab_data1' and 'tab_data2'
formats
tab_schema = {
"fields": [
{"name":"index", "type":"integer", "constraint":{"minimum":50}},
{"name":"dates", "type":"date"},
{"name":"value", "type":"integer"},
{"name":"coord", "type":"geopoint", "format":"array"},
{"name":"names"},
{"name":"unique", "type":"boolean"}
]
}
THOMY Expires 21 June 2024 [Page 29]
Internet-Draft NTV tabular format (NTV-TAB) December 2023
Figure 3: Schema example
Acknowledgements
TBD
Contributors
TBD
Author's Address
Philippe THOMY
Loco-labs
476 chemin du gaf de Famian
84 500 BOLLENE
France
Email: philippe@loco-labs.io
URI: https://github.com/loco-philippe/NTV/blob/main/README.md
THOMY Expires 21 June 2024 [Page 30]
