Binary Syntax for SIP Common Log Format

This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents in effect on the date of publication of this document (http://trustee.ietf.org/license-info). Please review these documents carefully, as they describe your rights and restrictions with respect to this document.

Abstract

This document proposes a binary syntax for the SIP common log format (CLF). It does not cover semantic issues, and is meant to be evaluated in the context of the other efforts discussing SIP CLF.

1. Introduction
2. Format
3. Example Record
4. Text Tool Considerations
5. Normative References
Appendix A. Acknowledgements
§ Author's Address

1. Introduction

This document counter-proposes a format that is no more difficult to generate by logging entities, while being radically faster to process. In particular, the format is optimized for both rapidly scanning through log records, as well as quickly locating commonly-accessed data fields. Both operations can be performed in constant time (as compared with O(n) time associated with the current format, where n is the length of the log record).

Further, the format proposed by this document retains the ability to be read by humans and processed using traditional Unix text processing tools, such as sed, awk, perl, cut, and grep.

2. Format

Each data record is encoded according to the following format. Note that indications of "hexadecimal encoded" indicate that the value is to be written out in human-readable base-16 numbers using the ASCII characters 0x30 through 0x39 and 0x41 through 0x46 ('0' through '9' and 'A' through 'F'). Similarly, indications of "decimal encoded" indicate that the value is to be written out in human readable base-10 number using the ASCII characters 0x30 through 0x39 ('0' through '9'). In both encodings, numbers always take up the number of bytes indicated, and are padded on the left with ASCII '0' characters to fill the entire space.

 0      7 8     15 16    23 24    31
 +--------+--------+--------+--------+
 |Version |       Flags Field        | 0 - 3
 +--------+--------+--------+--------+
 |  0x2C  |      Record Length       | 4 - 7
 +--------+--------+--------+--------+
 |   Record Length (cont)   |  0x2C  | 8 - 11
 +--------+--------+--------+--------+
 |     Server Txn Pointer (Hex)      | 12 - 15
 +--------+--------+--------+--------+
 |      Server Txn Length (Hex)      | 16 - 19
 +--------+--------+--------+--------+
 |     Client Txn Pointer (Hex)      | 20 - 23
 +--------+--------+--------+--------+
 |      Client Txn Length (Hex)      | 24 - 27
 +--------+--------+--------+--------+
 |       Method Pointer (Hex)        | 28 - 31
 +--------+--------+--------+--------+
 |        Method Length (Hex)        | 32 - 35
 +--------+--------+--------+--------+
 |      To Value Pointer (Hex)       | 36 - 39
 +--------+--------+--------+--------+
 |       To Value Length (Hex)       | 40 - 43
 +--------+--------+--------+--------+
 |       To Tag Pointer (Hex)        | 44 - 47
 +--------+--------+--------+--------+
 |        To Tag Length (Hex)        | 48 - 51
 +--------+--------+--------+--------+
 |     From Value Pointer (Hex)      | 52 - 55
 +--------+--------+--------+--------+
 |      From Value Length (Hex)      | 56 - 59
 +--------+--------+--------+--------+
 |      From Tag Pointer (Hex)       | 60 - 63
 +--------+--------+--------+--------+
 |       From Tag Length (Hex)       | 64 - 67
 +--------+--------+--------+--------+
 |       Call-Id Pointer (Hex)       | 68 - 71
 +--------+--------+--------+--------+
 |       Call-Id Length (Hex)        | 72 - 75
 +--------+--------+--------+--------+
 |      TLV Start Pointer (Hex)      | 76 - 79
 +--------+--------+--------+--------+
 |  0x0A  |                          | 80 - 83
 +--------+                          +
 |            Date/Time              | 84 - 87
 +                          +--------+
 |                          |  0x2E  | 88 - 91
 +--------+--------+--------+--------+
 |        Fractional Seconds         | 92 - 95
 +                 +--------+--------+
 |                 |  0x09  |        | 96 - 99
 +--------+--------+--------+        +
 |                                   | 100 - 103
 +               CSeq                +
 |                                   | 104 - 107
 +        +--------+--------+--------+
 |        |  0x09  |    Response     | 108 - 111
 +--------+--------+--------+--------+
 |Response|  0x09  |                 | 112 - 115
 +--------+--------+                 +
 |                                   |
 |                                   |
 |         Mandatory Fields          |
 |                                   |
 |                                   |
 +--------+--------+--------+--------+ \
 |  0x09  |        Tag (Hex)         |  \
 +--------+--------+--------+--------+   \
 |Tag,cont|  0x2C  |  Length (Hex)   |    \   Repeated as
 +--------+--------+--------+--------+     >  many times
 |  Length (cont)  |  0x2C  |        |    /   as necessary
 +--------+--------+--------+        +   /
 |              Value                |  /
 +--------+--------+--------+--------+ /
 |  0x0A  |
 +--------+

First, an 80-byte header indicates meta-data about the record. Note that the field lengths encoded in the header do not include the ASCII tab characters used to separate fields from each other.

Bytes 12 through 72 contain hexadecimal-encoded pointer/length pairs that point to the values of variable-length mandatory fields. The "Pointer" fields indicate absolute byte values within the record, and must be >= 103. They point to the start of the corresponding value within the "Mandatory Fields" area. The "Length" fields indicate the length of the corresponding value. The final pointer, "TLV Start Pointer," points to the ASCII Tab (0x09) character for the first entry in the Tag/Length/Value area; if no such entries are present, this value is set to zero.

Note that the "Length" fields do not include the tab delimiters between fields. Further note that there are no delimiters between these pointer/length values -- they are packed together as a single, 68-character hexadecimal encoded string.

Following the pointer/length pairs, several fixed-length fields are encoded. As before, all fields are completely filled, pre-pending values with '0' characters as necessary.

After the "Mandatory Fields" section, Tag/Length/Value groups appear zero or more times. The location within the log record is indicated by the "TLV Start Ptr" field. They are used to log information that is not mandatory for all messages (although specific TLVs are mandatory in request logs).

3. Example Record

The following demonstrates approximately how a single log record appears in a logging file. Due to internet-draft conventions, this log entry has been split into ten lines, instead of the two lines that actually appear in a log file; and the tab characters have been padded out using spaces to simulate their appearance in a text terminal.

ARor,000181,
0072000D00800000008100060088002000A9000600B0002500D6000A00E100270108

1241708241.308241	0000000187	000	7yuz67jhyi9-9	-
INVITE	Bob <sip:bob@biloxi.example.com>	314159
Alice <sip:alice@atlanta.example.com>	9fxced76sl
3848276298220188511@atlanta.example.com
0000,0034,<sip:alice@client.atlanta.example.com;transport=tcp>
0001,001A,sip:bob@biloxi.example.com
0002,000C,192.168.9.12

A uuencoded version of this log entry (without the changes required to format it for an internet-draft) follows.

begin 644 sip-clf.txt
M05)O<BPP,#`Q.#$L,#`W,C`P,$0P,#@P,#`P,#`P.#$P,#`V,#`X.#`P,C`P
M,$$Y,#`P-C`P0C`P,#(U,#!$-C`P,$$P,$4Q,#`R-S`Q,#@*,3(T,3<P.#(T
M,2XS,#@R-#$),#`P,#`P,#$X-PDP,#`)-WEU>C8W:FAY:3DM.0DM"4E.5DE4
M10E";V(@/'-I<#IB;V)`8FEL;WAI+F5X86UP;&4N8V]M/@DS,30Q-3D)06QI
M8V4@/'-I<#IA;&EC94!A=&QA;G1A+F5X86UP;&4N8V]M/@DY9GAC960W-G-L
M"3,X-#@R-S8R.3@R,C`Q.#@U,3%`871L86YT82YE>&%M<&QE+F-O;0DP,#`P
M+#`P,S0L/'-I<#IA;&EC94!C;&EE;G0N871L86YT82YE>&%M<&QE+F-O;3MT
M<F%N<W!O<G0]=&-P/@DP,#`Q+#`P,4$L<VEP.F)O8D!B:6QO>&DN97AA;7!L
=92YC;VT),#`P,BPP,#!#+#$Y,BXQ-C@N.2XQ,@H`
`
end

4. Text Tool Considerations

This format has been designed to allow text tools to easily process logs without needing to understand the indexing format. Index lines may be rapidly discarded by checking the first character of the line: index lines will always start with an alphabetical character, while field lines will start with a numerical character.

Within a field line, script tools can quickly split fields at the tab characters. The first 11 fields are positional, and the meaning of any subsequent fields can be determined by checking the first four characters of the field. Alternately, these non-positional fields can be located using a regular expression. For example, the "Request URI" in a request can be found by searching for the perl regex /\t0001,....,([^\t]*)/.

Note also that requests can be distinguished from responses by checking the third positional field -- for requests, it will always be set to "000"; any other value indicates a response.

5. Normative References

Appendix A. Acknowledgements

Tom Taylor suggested the technique of combining the length field structure from the binary format with the human-readable ASCII format to allow both rapid processing by advanced tools, and easy processing by simpler, text-centric tools. Dean Willis suggested the use of tab delimiters as a means to avoid the need to escape values within a field. Vijay Gurbani provided significant feedback, and wrote the original proof-of-concept program which was adapted to produce the examples in this document.

	Adam Roach
	Tekelec
	17210 Campbell Rd.
	Suite 250
	Dallas, TX 75252
	US
Email:	adam@nostrum.com

Binary Syntax for SIP Common Log Format
draft-roach-sipping-clf-syntax-01

Status of this Memo

Copyright Notice

Abstract

Table of Contents

1. Introduction

2. Format

3. Example Record

4. Text Tool Considerations

5. Normative References

Appendix A. Acknowledgements

Author's Address

Binary Syntax for SIP Common Log Formatdraft-roach-sipping-clf-syntax-01

Status of this Memo

Copyright Notice

Abstract

Table of Contents

1. Introduction

2. Format

3. Example Record

4. Text Tool Considerations

5. Normative References

Appendix A. Acknowledgements

Author's Address

Binary Syntax for SIP Common Log Format
draft-roach-sipping-clf-syntax-01