Lokad.Serialize 1.3.1

Lokad.Serialize

This package is a Source Generator used to automatically implement binary serialization and deserialization. Projects that use this source generator must also reference Lokad.Serialize.Runtime, which provides an interface ISerializable<T>, an attribute class SerializeAttribute, and a static class Serializer.

Two steps to make a type T serializable:

1. make it implement interface ISerializable<T>
2. make it partial and annotate it with the [Serialize] attribute.

It is then possible to use Serializer.Write and Serializer.Read to serialize and deserialize instances of T.

using Lokad.Serialize.Runtime;

[Serialize]
public partial record Car(string Brand, int Year) : ISerializable<Car>;

BinaryReader reader = // ...
var car = Serializer.Read<Car>(reader);

BinaryWriter writer = // ...
Serializer.Write(car, writer);

Supported types

Records with [Serialize]

Record types can be annotated with [Serialize]. It is not necessary to annotate their properties as well, but it can be done in order to apply serialization options to them. For example, to deal with a string property that is often null:

[Serialize] 
public partial record Car(
    string Brand,
    int Year,
    [property: Serialize(OftenNull = true)] 
    string? License) : ISerializable<Car>;

The order of serialization of the fields is the order in which they appear in the record constructor.

You can also ignore a record property entirely:

[Serialize]
public partial record Car(
    string Brand,
    [property: Serialize(Ignore = true)] int CachedHash,
    [property: Serialize(Ignore = true)] string? TransientNote) : ISerializable<Car>;

Ignored properties are not serialized at all. During deserialization, ignored constructor arguments are supplied as default for value types and null for nullable reference types.

Plain data types with [Serialize]

A non-record class or struct can be annotated with [Serialize] if it has a parameter-less constructor. All the fields that are not readonly, const or static, and all the properties that support both get and either init or set will be considered for serialization. Two additional rules apply:

• All the considered fields and properties must be defined in the same file (so, no partial classes spreading their serializable members across multiple files) ; the order of serialization is the order of appearance in the file.
• All the considered fields and properties must be annotated with [Serialize] (to avoid unintentional serialization).

Note that the accessibility of the serialized members does not matter, and private members will be serialized as well as public ones.

For example:

[Serialize]
public partial class Car : ISerializable<Car>
{
    [Serialize] public required string Brand { get; init; }
    [Serialize] public required string Year { get; init; }
}

Plain data types take more effort than records, but are more flexible.

On plain data types, [Serialize(Ignore = true)] can be used on properties to keep them out of the wire format entirely. No property assignment is emitted during deserialization, so the property keeps the value established by the parameterless constructor or by its own initializer.

Interfaces and abstract classes with [Union]

An interface or abstract class can be made serializable by annotating it with [Union] to list all its supported sub-types and give them an integer tag.

Obviously, all sub-types must implement the interface or extend the abstract class in question, as well as implement ISerializable<>.

[Union(0, typeof(Cat))]
[Union(1, typeof(Dog))]
public partial interface IAnimal : ISerializable<IAnimal>
{
    public string Name { get; }
}

[Serialize]
public partial record Cat(
    string Name) : IAnimal, ISerializable<Cat>;

[Serialize] 
public partial class Dog : IAnimal, ISerializable<Dog>
{
    public required string Name { get; init; }
}

Basic types

The following types are supported "out of the box" as members of types annotated with [Serialize]:

• All primitive numeric types: byte, sbyte, short, ushort, int, uint, long, ulong, float, double and decimal
• Booleans: bool (serialized as a single bit, if possible)
• Text: string (as UTF-8) and char (treated as a single-character string)
• Other basic types: DateTime
• Enums (serialized as their underlying type)
• Nullable: System.Nullable<T> if T is a serializable struct or basic value type.
• Collection types (so long as the element types are themselves serializable or basic types): T[], List<T>, IReadOnlyList<T>, IReadOnlyCollection<T>, HashSet<T>, Dictionary<TK, TV> and IReadOnlyDictionary<TK, TV>

Blittable types

Unmanaged structs can be marked as blittable, which means they are serialized by copying their bytes directly.

Use [Serialize(Blittable = true)] to serialize it by blitting. The member type must be an unmanaged struct, or an array or ReadOnlyMemory or Nullable<T> of an unmanaged struct.

public struct Point
{
    public int X;
    public int Y;
}

[Serialize]
public partial class Sample : ISerializable<Sample>
{
    [Serialize(Blittable = true)] public Point A { get; set; }
    [Serialize(Blittable = true)] public Point? B { get; set; }
    [Serialize(Blittable = true)] public Point[] C { get; set; } = [];
    [Serialize(Blittable = true)] public ReadOnlyMemory<Point> D { get; set; }
}

In order to annotate as blittable a type that appears in another case, such as in an IReadOnlyList or IReadOnlyDictionary, use the [Serialize(Blittables = [...])] annotation instead to list types that should be treated as blittable. This can be done at the type level (applies to all properties and fields of the type) or on an individual member (applies only to the member).

public struct Point
{
    public int X;
    public int Y;
}

public struct Size
{
    public int W;
    public int H;
}

[Serialize(Blittables = new[] 
{
    typeof(Point), 
    typeof(Size)
})]
public partial class Canvas : ISerializable<Canvas>
{
    [Serialize] public Point Origin { get; set; }
    [Serialize] public Size Extent { get; set; }
}

You can also flag a type as always-blittable by using [Serialize(Blittable = true)] on it.

[Serialize(Blittable = true)]
public struct Point
{
    public int X;
    public int Y;
}

[Serialize]
public partial class Sample : ISerializable<Sample>
{
    [Serialize] public Point A { get; set; }
    [Serialize] public IReadOnlyDictionary<string, Point> B { get; set; }
}

Serialization format

Lokad.Serialize uses a combination of three binary serialization formats. Which format is used depends on the types and annotations.

Marshal serialization

This is the lowest level, and is used for serializing unmanaged types, including collections containing only unmanaged types. In this serialization format, all values are read and written by copying the exact bytes of the value (as with System.Runtime.InteropServices.MemoryMarshal).

This serialization is used:

• When annotating with [Serialize(Blittable = true)] or [Serialize(Blittables = ...)].
• For all numeric primitives (so, all primitive types except string, char and bool) as well as numeric-compatible primitives (System.DateTime, enums).
• For bool in complex types such as bool? or Dictionary<T, bool>, saving it as a full byte.

There is no backwards compatibility for this serialization mode: the type must be read with exactly the same fields as it was written with.

BinaryWriter serialization

This serialization mode uses (or mirrors) the System.IO.BinaryWriter and System.IO.BinaryReader classes. For most types, it behaves like Marshal serialization. There are two special cases:

LEB128 serialization stores integers in fewer bytes than usual (one byte for 0..127, two bytes for 128..16383, and so on). This serialization takes more space when used for negative numbers!

Characters are encoded as a single UTF-8 code point.

String serialization stores an array of bytes corresponding to the UTF-8 encoding of the string, preceded by the number of bytes as a LEB128-encoded integer.

Classes and structs are saved by concatenating the serialization of their individual fields.

Arrays and collections are stored as a number of entries (a LEB128-encoded integer), followed by the concatenation of the elements.

This serialization is used:

• LEB128 serialization, for all multi-byte numeric types annotated with [Serialize(OftenSmall = true)], such as short, ushort, int, uint, long, ulong, and enums not backed as byte.
• Character serialization, for char
• String serialization, for string, ReadOnlyMemory<char>, Memory<char>, IReadOnlyList<char>, List<char>.
• Class/struct serialization, for all types that don't require trigger Tagged serialization (see below), including Tuple<...> and ValueTuple<...> (of up to 7 elements), and KeyValuePair<,>.
• Collection serialization, for T[], ReadOnlyMemory<T>, Memory<T>, ReadOnlyMemory<T>, IReadOnlyList<T>, List<T>, HashSet<T>, IReadOnlyDictionary<K,V> and Dictionary<K,V> (the latter two being treated as collections of KeyValuePair<K,V>)

Note that for unordered collections of types with varying hashcodes (such as HashSet<string> or Dictionary<string, T>), the same code producing the same collection may still lead to different serialization results, since the traversal order can change from one execution to the next.

There is no backwards compatibility for this serialization mode: the type must be read with exactly the same fields as it was written with.

Tagged serialization

This serialization mode prefixes classes and structs with a variable-length header that enables backwards compatibility. The binary structure of this header is:

• The first byte b0 contains one "flags-or-union" discriminator bit b0 & 1, will be 1 to indicate a union tag, or 0 to indicate a flag-set header.
• Every byte b contains one "continues" discriminator bit b0 & 0x80, will be 1 to indicate that another header byte follows, or 0 to indicate this is the last header byte.
• The remaining bits (six in the first byte, seven in the others) are the header payload.

When reading a union type (such as an interface or abstract class), the header is almost always a union discriminator, the payload is interpreted as an integer and used to select one of the union members. This header is then followed by the union member's own serialization.

If the union type has a default case (by setting to true the third argument, as in [Union(42, typeof(T), true)]) then the header is allowed to not be a union header ; the default case is then used, and the payload is taken to be part of the default case's serialization instead of the union type. The intent is backwards compatibility: if a value was previously serialized as a Cat field (in a class or struct), it can still be properly deserialized if the type of the field is changed to IAnimal, so long as Cat is the default union case for IAnimal.

When reading another type (such as a class, struct or record) that has optional fields, each bit controls the presence or absence of that field. If the header is too short, the missing bits are assumed to be zero, indicating the absence of the fields. Again, the intent is backwards compatibility, where a new optional field can be added to a tagged type, and old data without the field can still be deserialized (with the field becoming null).

To specify optional fields:

• bool fields are always optional, default is false.
• Fields of nullable types (both System.Nullable and reference types) are always optional, default is null.
• Value types can be made optional by annotating them with [Serialize(OftenDefault = true)], default is default for that type.
• Collection types can be made optional by annotating them with [Serialize(OftenEmpty = true)], default is an empty collection.
• Properties can be excluded entirely by annotating them with [Serialize(Ignore = true)]. This is allowed on value types, nullable reference types, and on non-required properties. Ignored properties are neither serialized nor deserialized.

Adding a new optional field to a class, after the existing fields, is always backwards-compatible if the class previously had optional fields (or if it is annotated as [Serialize(FutureExtensions = true)] with or without optional fields).

Changing the optional status of an existing field, however, is not backwards-compatible.

Finally, if the last field T Last of a type is not optional, then it is backwards compatible to annotate it as [Serialize(OftenSingle = true)] and change its type to a collection of T. Previously serialized values will be loaded as a single-element collection.

Span-based serialization

As an alternative to BinaryReader and BinaryWriter, you can work with spans using SpanReader and SpanWriter. Both are created from an entire span, so it is not possible to load in additional data from another source when reading, or to have an expanding destination span when writing.

using Lokad.Serialize.Runtime;

[Serialize]
public partial record Car(string Brand, int Year) : ISerializable<Car>;

var destination = new byte[1024];
var writer = new SpanWriter(destination.AsSpan());
Serializer.Write(car, writer);

// Use writer.Position to know how many bytes were written.
var serialized = destination.AsSpan(0, writer.Position);

var reader = new SpanReader(serialized);
var car = Serializer.Read<Car>(reader);

Attempting to read or write past the end of the span will throw an IndexOutOfRangeException.