Knapcode.TNetstrings 0.2.0

Knapcode.TNetstrings

This is a .NET library for processing tnetstrings data. TNetstrings is a tagged data format that puts the data type after the data, which means serialization and deserialization often require the data to fully reside in memory to make sense of it.

TNetstrings have 7 data types.

• Bytes (called "strings" by the specification)
• Integer
• Float
• Boolean
• Null
• Dictionary
• List

The mitmproxy project adds an additional type of UTF-8 strings. This is optionally supported by this library (off by default to be compliant with the tnetstrings specification).

A simple dictionary looks like this:

30:1:a,1:1#2:bb,2:22#3:ccc,3:333#}

This is logically equivalent to the following JSON:

{
    "a": 1,
    "bb": 22,
    "ccc": 333
}

The serialization is done with TNetstringWriter. It writes to an IBufferWriter<byte> instance. One easy way to get an IBufferWriter<byte> instance is with a System.IO.Pipelines.PipeWriter, e.g. PipeWriter.Create(myWriteableStream). PipeWriter implements IBufferWriter<byte>.

The deserialization is done with TNetstringReader. It reads from a System.IO.Pipelines.PipeReader, which can be created from a stream using PipeReader.Create(myReadableStream). Call ReadNextAsync() on the TNetstringReader until false is returned. For each call, the Type and Data properties will be populated. This reader does not recurse into structures like the list or dictionary. Instead, it is meant for iterating over concatenated tnetstrings. To process each record, use TNetStringDataReader or use the ParseData method on TNetstringReader.

Samples

Serialize a dictionary

Serialize a dictionary into a tnetstring.

var bufferWriter = new ArrayBufferWriter<byte>();
var writer = new TNetstringWriter(bufferWriter);
IDictionary dict = new Dictionary<string, int>
{
    { "a", 1 },
    { "bb", 22 },
    { "ccc", 333 }
};
writer.DumpDictionary(dict);
string result = Encoding.UTF8.GetString(bufferWriter.WrittenSpan);
Assert.Equal("30:1:a,1:1#2:bb,2:22#3:ccc,3:333#}", result);

Serialize a list

Serialize a list into a tnetstring.

var bufferWriter = new ArrayBufferWriter<byte>();
var writer = new TNetstringWriter(bufferWriter);
IEnumerable list = new List<string> { "a", "bb", "ccc" };
writer.DumpList(list);
string result = Encoding.UTF8.GetString(bufferWriter.WrittenSpan);
Assert.Equal("15:1:a,2:bb,3:ccc,]", result);

Streaming deserialization

Process concatenated tnetstrings.

using var stream = File.OpenRead(path);
var pipeReader = PipeReader.Create(stream);
using var reader = new TNetstringReader(pipeReader, new TNetstringReaderOptions
{
     // allow the ';' (UTF-8 string) type extension
    AllowUtf8Strings = true,

    // return dictionaries as Dictionary<string, object?>
    DictionaryKeysAsStrings = true, 

    // try to decode all bytes as strings
    ByteStringEncoding = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true),
});

while (await reader.ReadNextAsync())
{
    Console.WriteLine($"Type: {reader.Type}, length: {reader.Data.Length}");
    var result = reader.ParseData();
    Console.WriteLine("Parsed the data!");
}

Data deserialization

Once you have a data value in memory, you can use TNetStringDataReader to process it. This is a readonly ref struct and expects only the data portion of the tnetstring (not the length prefix or type suffix).

// no length prefix (implied by the size of the bytes passed in)
// no type suffix (implied by the Read* method called on the reader)
var tnet = "3:foo,3:bar,3:baz,";
var bytes = Encoding.UTF8.GetBytes(tnet);
var reader = new TNetstringDataReader(bytes);
var list = reader.ReadList();
Assert.Equal(new object?[] { "foo", "bar", "baz" }, list.Cast<byte[]>().Select(Encoding.UTF8.GetString).ToList());