MASES.KNet.Serialization.Avro 2.7.5

dotnet add package MASES.KNet.Serialization.Avro --version 2.7.5                
NuGet\Install-Package MASES.KNet.Serialization.Avro -Version 2.7.5                
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="MASES.KNet.Serialization.Avro" Version="2.7.5" />                
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add MASES.KNet.Serialization.Avro --version 2.7.5                
#r "nuget: MASES.KNet.Serialization.Avro, 2.7.5"                
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install MASES.KNet.Serialization.Avro as a Cake Addin
#addin nuget:?package=MASES.KNet.Serialization.Avro&version=2.7.5

// Install MASES.KNet.Serialization.Avro as a Cake Tool
#tool nuget:?package=MASES.KNet.Serialization.Avro&version=2.7.5                

title: Serialization usage of .NET suite for Apache Kafka _description: Describes how to use serialization of .NET suite for Apache Kafka

KNet: Serializer/Deserializer

KNet comes with a base set of serializer and deserializer. Most of them are usable with primitives types (bool, int, etc) and array of bytes.

If the user wants to use structures types there are two ways:

  1. Creates types in Java and reflects them in C#
  2. Creates types in C# and extend the available serializer/deserializer

KNet suite offers some ready made serializer/deserializer usable with the specific APIs (KNetProducer/KNetConsumer).

The current available packages are:

Starting from version 2.7.0, KNet comes with two kind of data exchange mechanisms:

  • Raw: data exchanges, across JVM-CLR boundary, using byte array transfer and this is the standard used since last version
  • Buffered: data exchanges, across JVM-CLR boundary, using ByteBuffer objects:
    • this version avoids a real data move, only references to ByteBuffer are exchanged
    • the serializer/deserializer shares, with the ByteBuffer, the memory pointers originating the information and the counterpart reads that memory without make copies

All available packages listed at the beginning comes with both versions and the user can choose its preferred one.

As example, let consider a type defined like the following one:

public class TestType
{
    public TestType(int i)
    {
        name = description = value = i.ToString();
    }

    public string name;
    public string description;
    public string value;

    public override string ToString()
    {
        return $"name {name} - description {description} - value {value}";
    }
}

To manage it within C#, without create TestType in Java, an user can create:

  • serializer (the body must be updated with the user serializer):
SerDesRaw<TestType> serializer = new SerDesRaw<TestType>()
{
    OnSerialize = (topic, type) => { return Array.Empty<byte>(); }
};
  • deserializer (the body must be updated with the user deserializer):
SerDesRaw<TestType> deserializer = new SerDesRaw<TestType>()
{
    OnDeserialize = (topic, data) => { return new TestType(0); }
};

Otherwise the user can use a ready made class like in the following snippet:

ISerDesRaw<TestType> serdes = JsonSerDes.Value<TestType>.NewByteArraySerDes();

A single JsonSerDes.ValueRaw can be used in serialization and deserialization, and produce Json serialized data.

Key and Value versions

The reader noticed that in the example was used JsonSerDes.Value<T>().NewByteArraySerDes(). It is a serializer/deserializer, based on byte array, generally used for values because it stores, within the record Headers information related to the value itself.

All packages listed above have multiple types based on the scope and data exchange mechanism:

  • [Serialization Format].Key: key serializer/deserializer can manages data transfer using both byte array and ByteBuffer
  • [Serialization Format].Value: value serializer/deserializer can manages data transfer using both byte array and ByteBuffer

where [Serialization format] depends on the serializatin package in use and the selection of the data transfer can be made from underlying code or can be requested from the user:

  • [Serialization Format].[Key or Value]<TData>.NewByteArraySerDes(): returns an ISerDesRaw<TData>
  • [Serialization Format].[Key or Value]<TData>.NewByteBufferSerDes(): returns an ISerDesBuffered<TData>

[!TIP] As specified above, each serializer stores info within the Headers and this behavior is controlled from a property named UseHeaders. If the user writes a code like:

ISerDesRaw<TestType> serdes = JsonSerDes.Value<TestType>.NewByteArraySerDes();
serdes.UseHeader = false;

The ISerDesRaw<TestType> instance does not writes the Headers and can be used both for key and value.

Specific cases

Some kind of serializers extension have specific needs will be listed below.

Avro serializer

The Avro serializer is based on Apache.Avro package. The types managed are:

  • Avro types managed using the Avro library are Avro records which:

NOTE: simple types (the one that have an Apache Kafka default serializer) are not managed and will be refused

MessagePack serializer

The MessagePack serializer is based on MessagePack package. The types managed are:

  • MessagePack types managed using the MessagePack library shall be MessagePack types.

NOTE: simple types (the one that have an Apche Kafka default serializer) are not managed and will be refused

Protobuf serializer

The Protobuf serializer is based on Google.Protobuf package. The types managed are:

  • Protobuf types managed using the Protobuf library shall be messages types which:
    • Shall have a parameterless constructor
    • Shall conform to IMessage<T>

NOTE: simple types (the one that have an Apche Kafka default serializer) are not managed and will be refused

Product Compatible and additional computed target framework versions.
.NET net6.0 is compatible.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
.NET Framework net462 is compatible.  net463 was computed.  net47 was computed.  net471 was computed.  net472 was computed.  net48 was computed.  net481 was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
2.7.5 138 7/2/2024
2.7.4 157 6/27/2024
2.7.3 177 6/24/2024
2.7.2 137 5/25/2024
2.7.1 104 5/18/2024
2.7.0 87 5/16/2024
2.6.4 79 6/14/2024
2.6.3 68 6/11/2024
2.6.2 65 5/17/2024
2.6.1 100 5/3/2024
2.6.0 150 3/1/2024
2.5.0 109 2/28/2024
2.4.3 109 2/11/2024
2.4.2 109 1/27/2024
2.4.1 117 1/21/2024
2.4.0 80 1/20/2024
2.3.0 231 11/25/2023
2.2.0 438 10/19/2023
2.1.3 110 10/11/2023
2.1.2 115 10/6/2023
2.1.1 106 10/5/2023
2.1.0 102 9/27/2023
2.0.2 152 8/2/2023
2.0.1 138 7/11/2023
2.0.0 146 7/8/2023
1.5.5 138 7/1/2023
1.5.4 140 5/28/2023