FluentSync 1.2.0

A .Net library with a fluent interface for comparing and synchronizing entities/records.

Overview

Have you ever wanted to synchronize data between multiple databases, systems, a database and an API, etc.? Have you ever wanted to resync data between multiple systems because something went wrong and some data is no longer up-to-date? If the answer is Yes then this library is for you.

Get Started

FluentSync can be installed using the NuGet package manager or the dotnet CLI.

Install-Package FluentSync

dotnet add package FluentSync

Why FluentSync

• Targets .Net Standard 2.1 and .Net Framework 4.6.2
• Fluent interface
• No dependencies on other libraries
• 236 tests
• ~98% line coverage

Usage

The FluentSync library has Comparer agents and Sync agents. The Comparer agent compares the source and destination items/entities. Then the Sync agent uses this comparison result to determine which items/entities will be inserted/updated/deleted in the source and destination according to the sync configurations.

Sync modes:

• TwoWay: updates the source and the destination to have the same items.
• MirrorToDestination: updates the destination only to have the same items as in the source, and deletes any item in the destination that does not have a match.
• MirrorToSource: updates the source only to have the same items as in the destination, and deletes any item in the source that does not have a match.
• UpdateDestination: updates the destination items only to have the same items as in the source. It does not delete any item from the destination and it does not update conflicts.
• UpdateSource: updates the source items only to have the same items as in the destination. It does not delete any item from the source and it does not update conflicts.
• Custom: you can have your own custom sync mode by customizing the properties of the SyncMode class.

ComparerAgent Usage

Basically, you can compare two lists of any type such as a string with the following code:

List<string> source = new List<string> { "Tom", "Tim", "bob", "Zoo" }
	, destination = new List<string> { "Bob", "Sam", "Tim" };

var comparisonResult = await ComparerAgent<string>.Create()
	.SetSourceProvider(source)
	.SetDestinationProvider(destination)
	.CompareAsync(CancellationToken.None).ConfigureAwait(false);

// This is the comparison result which is verified by the FluentAssertions library
comparisonResult.ItemsInSourceOnly.Should().BeEquivalentTo(new List<string> { "Tom", "bob", "Zoo" });
comparisonResult.ItemsInDestinationOnly.Should().BeEquivalentTo(new List<string> { "Bob", "Sam" });

comparisonResult.Matches.Should().BeEquivalentTo(new List<MatchComparisonResult<string>>
{
	new MatchComparisonResult<string>{Source = "Tim", Destination = "Tim", ComparisonResult = MatchComparisonResultType.Same}
});

You can also have a custom key selector for the ComparerAgent to identify each item.

List<string> source = new List<string> { "Tom", "Tim", "bob", "Zoo" }
	, destination = new List<string> { "Bob", "Sam", "Tim" };

var comparisonResult = await ComparerAgent<string>.Create()
	.SetKeySelector(x => x?.ToLower())
	.SetSourceProvider(source)
	.SetDestinationProvider(destination)
	.CompareAsync(CancellationToken.None).ConfigureAwait(false);

// This is the comparison result
comparisonResult.ItemsInSourceOnly.Should().BeEquivalentTo(new List<string> { "Tom", "Zoo" });
comparisonResult.ItemsInDestinationOnly.Should().BeEquivalentTo(new List<string> { "Sam" });

comparisonResult.Matches.Should().BeEquivalentTo(new List<MatchComparisonResult<string>>
{
	new MatchComparisonResult<string>{Source = "Tim", Destination = "Tim", ComparisonResult = MatchComparisonResultType.Same},
	new MatchComparisonResult<string>{Source = "bob", Destination = "Bob", ComparisonResult = MatchComparisonResultType.Conflict}
});

Let's compare entities and determine which entities are newer if they exist in the source and destination. The source/destination provider for the comparer agent is a provider that implements the IComparerProvider<TItem> interface which gets the items/entities; passing an IEnumerable<TItem> directly, as in the examples above, wraps it in one for you.

var comparisonResult = await ComparerAgent<int?, Event>.Create()
	.SetKeySelector(e => e.Id)
	.SetCompareItemFunc((s, d) =>
	{
		if (s.Title == d.Title && s.ModifiedDate == d.ModifiedDate)
			return MatchComparisonResultType.Same;
		else if (s.ModifiedDate < d.ModifiedDate)
			return MatchComparisonResultType.NewerDestination;
		else if (s.ModifiedDate > d.ModifiedDate)
			return MatchComparisonResultType.NewerSource;
		else
			return MatchComparisonResultType.Conflict;
	})
	.SetSourceProvider(source)
	.SetDestinationProvider(destination)
	.CompareAsync(CancellationToken.None).ConfigureAwait(false);

Do you have a compound primary key for the entity? Yes. No problem, it is supported by using Tuple.

var comparisonResult = await ComparerAgent<Tuple<int?, int?>, PersonHobby>.Create()
	.SetKeySelector(x => new Tuple<int?, int?>(x.PersonId, x.HobbyId))
	.SetCompareItemFunc((s, d) => (s.PersonId == d.PersonId && s.HobbyId == d.HobbyId && s.LoveScale == d.LoveScale) ? MatchComparisonResultType.Same : MatchComparisonResultType.Conflict)
	.SetSourceProvider(source)
	.SetDestinationProvider(destination)
	.CompareAsync(CancellationToken.None).ConfigureAwait(false);

If you only need to know which keys exist on each side, use the KeyComparerAgent. It compares keys instead of whole entities and returns a KeysComparisonResult<TKey> with KeysInSourceOnly, KeysInDestinationOnly, and Matches. This is the agent the BatchSyncAgent uses internally. Unlike the ComparerAgent it has no configurations: a null key or a repeated key on either side always throws.

var keysComparisonResult = await KeyComparerAgent<int>.Create()
	.SetSourceProvider(new List<int> { 5, 4, 9 })
	.SetDestinationProvider(new List<int> { 6, 10, 5 })
	.CompareAsync(CancellationToken.None).ConfigureAwait(false);

By default the ComparerAgent accepts duplicate keys, duplicate items, and null items on both sides. You can tighten that through the agent's configurations, which throw when the rule is violated. Each rule is set per side with RuleAllowanceType: Both (the default), Source, Destination, or None, where the value names the side the rule is allowed on.

var comparerAgent = ComparerAgent<string>.Create()
	.Configure((c) =>
	{
		c.AllowDuplicateKeys = RuleAllowanceType.None;
		c.AllowNullableItems = RuleAllowanceType.Destination; // Source items must not be null
	});

For validation the built-in rules don't cover, SetValidateItemsAction runs your own check against both sides after the items are loaded and before they are compared. Throw from it to abort the comparison.

var comparerAgent = ComparerAgent<int?, Person>.Create()
	.SetKeySelector(x => x.Id)
	.SetValidateItemsAction((source, destination) =>
	{
		if (source.Any(x => x.Id < 0))
			throw new ArgumentException("Negative Ids are invalid in the source list.");
	});

SyncAgent Usage

Simply, you can compare and sync the two lists with the following code:

List<int> source = new List<int> { 5, 4, 9 }
	, destination = new List<int> { 6, 10, 5 };

await SyncAgent<int>.Create()
	.Configure((c) => c.SyncMode.SyncModePreset = SyncModePreset.MirrorToDestination)
	.SetComparerAgent(ComparerAgent<int>.Create())
	.SetSourceProvider(source)
	.SetDestinationProvider(destination)
	.SyncAsync(CancellationToken.None).ConfigureAwait(false);

// This code is here to verify the two lists
source.Should().BeEquivalentTo(new List<int> { 5, 4, 9 });
destination.Should().BeEquivalentTo(new List<int> { 5, 4, 9 });

Another example of the sync agent with a custom key selector and an item comparer function.

await SyncAgent<string>.Create()
	.Configure((c) => c.SyncMode.SyncModePreset = SyncModePreset.TwoWay)
	.SetComparerAgent(ComparerAgent<string>.Create()
		.SetKeySelector(x => x?.ToLower())
		.SetCompareItemFunc((s, d) => s == d ? MatchComparisonResultType.Same : MatchComparisonResultType.Conflict))
	.SetSourceProvider(sourceItems)
	.SetDestinationProvider(destinationItems)
	.SyncAsync(CancellationToken.None).ConfigureAwait(false);

Call order matters: SetSourceProvider and SetDestinationProvider also wire the provider into the comparer agent, so SetComparerAgent has to come first. Calling them the other way round throws a NullReferenceException telling you the ComparerAgent must be set first. The same applies to the BatchSyncAgent, where the dictionary overloads additionally need SetKeySelector to have been called.

You can also compare and sync entities. The source/destination provider for the sync agent is a provider that implements the IComparerSyncProvider<TItem> interface, which both gets the items/entities and does the CRUD operations for them. Passing an IList<TItem> or a SortedSet<TItem> directly, as in the examples above, wraps it in a built-in ListSyncProvider<TItem> or SortedSetSyncProvider<TItem>.

await SyncAgent<int?, Event>.Create()
	.Configure((c) => c.SyncMode.ItemsInSourceOnly = SyncItemOperation.Add) // Custom SyncMode
	.SetComparerAgent(ComparerAgent<int?, Event>.Create()
	.SetKeySelector(x => x.Id)
	.SetCompareItemFunc((s, d) =>
	{
		if (s.Title == d.Title && s.ModifiedDate == d.ModifiedDate)
			return MatchComparisonResultType.Same;
		else if (s.ModifiedDate < d.ModifiedDate)
			return MatchComparisonResultType.NewerDestination;
		else if (s.ModifiedDate > d.ModifiedDate)
			return MatchComparisonResultType.NewerSource;
		else
			return MatchComparisonResultType.Conflict;
	}))
	.SetSourceProvider(source)
	.SetDestinationProvider(destination)
	.SyncAsync(CancellationToken.None).ConfigureAwait(false);

Important note: the SyncAgent loads all the entities in memory in one call, then compares them, and finally synchronizes the entities. It is not recommended to use this agent if you have thousands of entities, use BatchSyncAgent instead.

BatchSyncAgent Usage

It is highly recommended to use the BatchSyncAgent to synchronize the entities if you have thousands of entities; it synchronizes them in batches. The source/destination provider for the batch sync agent is a provider that implements the IComparerBatchSyncProvider<TKey, TItem> interface, which gets the keys, gets the entities by their keys, and does the CRUD operations for them. Passing an IDictionary<TKey, TItem> directly wraps it in the built-in DictionaryBatchSyncProvider<TKey, TItem>. The default batch size is 100.

await BatchSyncAgent<int, Person>.Create()
	.Configure((c) =>
	{
		c.SyncMode.SyncModePreset = SyncModePreset.UpdateDestination;
		c.BatchSize = 50;
	})
	.SetComparerAgent(KeyComparerAgent<int>.Create())
	.SetKeySelector(x => x.BusinessEntityID)
	.SetCompareItemFunc((s, d) =>
	{
		if (s.FirstName == d.FirstName
			&& s.LastName == d.LastName
			&& s.ModifiedDate == d.ModifiedDate)
			return MatchComparisonResultType.Same;
		else if (s.ModifiedDate < d.ModifiedDate)
			return MatchComparisonResultType.NewerDestination;
		else if (s.ModifiedDate > d.ModifiedDate)
			return MatchComparisonResultType.NewerSource;
		else
			return MatchComparisonResultType.Conflict;
	})
	.SetSourceProvider(sourceProvider)
	.SetDestinationProvider(destinationProvider)
	// You can use these actions to show the progress on the screen or in a log file.
	.SetBeforeSyncingKeysAction((cr) => Console.WriteLine($"Before syncing persons > persons in source only: {cr.KeysInSourceOnly.Count}, persons in destination only: {cr.KeysInDestinationOnly.Count}, persons in source and destination: {cr.Matches.Count}"))
	.SetBeforeSyncingAction((cr) => Console.WriteLine($"Syncing persons batch > add: {cr.ItemsInSourceOnly.Count}, remove: {cr.ItemsInDestinationOnly.Count}, update: {cr.Matches.Count(x => x.ComparisonResult != MatchComparisonResultType.Same)}, Same: {cr.Matches.Count(x => x.ComparisonResult == MatchComparisonResultType.Same)}"))
	.SetBeforeDeletingItemsFromSourceAction((list) => Console.WriteLine($"Deleting from source {list.Count} persons"))
	.SetBeforeDeletingItemsFromDestinationAction((list) => Console.WriteLine($"Deleting from destination {list.Count} persons"))
	.SyncAsync(cancellationToken);

Important note: the BatchSyncAgent loads all the keys in memory in one call at the beginning, which consumes less memory than loading all entities, then compares the keys only. After that it loads the batch entities by the keys and compares them, and finally syncs the entities.

Customizing the Sync Mode

Each preset is shorthand for five properties on the SyncMode class, which you can also set individually for a Custom sync mode. Items without a match are governed by a SyncItemOperation (None, Add, or Delete), and matched pairs by a SyncMatchOperation (None, UpdateSource, UpdateDestination, UpdateOldItem, UpdateOldDestination, or UpdateOldSource).

Two things to know before mixing these with a preset:

• Assigning SyncModePreset overwrites all five properties, and assigning Custom (like None) resets them all to "do nothing". Set the preset first, then customize, as in the SyncAgent example above.
• ConflictMatches rejects UpdateOldItem, UpdateOldDestination, and UpdateOldSource — with neither side known to be newer there is no "old" item to update — and throws if you assign one.

await SyncAgent<int>.Create()
	.Configure((c) =>
	{
		c.SyncMode.SyncModePreset = SyncModePreset.TwoWay; // Start from a preset
		c.SyncMode.ItemsInDestinationOnly = SyncItemOperation.Delete; // Then adjust
	})
	.SetComparerAgent(ComparerAgent<int>.Create())
	.SetSourceProvider(source)
	.SetDestinationProvider(destination)
	.SyncAsync(CancellationToken.None).ConfigureAwait(false);

The order of the operations is configurable too. SyncOperationsOrder.Order controls whether inserts, updates, or deletes are applied first and defaults to Delete, Update, Insert. For the BatchSyncAgent, BatchSyncListsOrder.Order additionally controls which of the three lists is synced first and defaults to ItemsInSourceOnly, ItemsInDestinationOnly, Matches. Neither array may contain duplicates; the agent validates this and throws before syncing.

await BatchSyncAgent<int, Person>.Create()
	.Configure((c) =>
	{
		c.SyncMode.SyncModePreset = SyncModePreset.MirrorToDestination;
		c.SyncOperationsOrder.Order[0] = SyncOperationType.Insert; // Insert before deleting
		c.SyncOperationsOrder.Order[2] = SyncOperationType.Delete;
		c.BatchSyncListsOrder.Order[0] = BatchSyncListType.Matches; // Sync the matches first
		c.BatchSyncListsOrder.Order[2] = BatchSyncListType.ItemsInSourceOnly;
	})
	.SetComparerAgent(KeyComparerAgent<int>.Create())
	.SetKeySelector(x => x.BusinessEntityID)
	.SetCompareItemFunc((s, d) => s.ModifiedDate == d.ModifiedDate ? MatchComparisonResultType.Same : MatchComparisonResultType.Conflict)
	.SetSourceProvider(sourceProvider)
	.SetDestinationProvider(destinationProvider)
	.SyncAsync(cancellationToken);

More Examples

For more examples you can look at the unit tests project FluentSync.Tests.

Building and Testing

The library targets .Net Standard 2.1 and .Net Framework 4.6.2; the test and benchmark projects target .Net 10, so building the solution needs the .Net 10 SDK.

dotnet build -c Release
dotnet test

Running Benchmarks

Performance benchmarks live in the FluentSync.Benchmarks project and use BenchmarkDotNet. They cover ComparerAgent, KeyComparerAgent, SyncAgent, and BatchSyncAgent at 1k / 10k / 100k input sizes and report mean time plus allocations.

Benchmarks must be run in Release configuration — BenchmarkDotNet refuses to run a debug build.

# Run everything
dotnet run -c Release --project FluentSync.Benchmarks -- --filter *

# Run a single agent's benchmarks
dotnet run -c Release --project FluentSync.Benchmarks -- --filter *ComparerAgentBenchmarks*

# Quick smoke run (one warmup and one iteration per case; checks that everything runs, timings are not meaningful)
dotnet run -c Release --project FluentSync.Benchmarks -- --filter * --job Dry

Results are written to BenchmarkDotNet.Artifacts/ in the directory you run the command from (the repository root for the commands above). To capture a baseline before optimizing, save a copy of that folder and diff against a re-run after your changes.

License

FluentSync is licensed under the Apache License 2.0.