MVFC.DataX.Core
1.2.0
dotnet add package MVFC.DataX.Core --version 1.2.0
NuGet\Install-Package MVFC.DataX.Core -Version 1.2.0
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="MVFC.DataX.Core" Version="1.2.0" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="MVFC.DataX.Core" Version="1.2.0" />
<PackageReference Include="MVFC.DataX.Core" />
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add MVFC.DataX.Core --version 1.2.0
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
#r "nuget: MVFC.DataX.Core, 1.2.0"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#:package MVFC.DataX.Core@1.2.0
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package.
#addin nuget:?package=MVFC.DataX.Core&version=1.2.0
#tool nuget:?package=MVFC.DataX.Core&version=1.2.0
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
MVFC.DataX.Core
Core abstractions and interfaces (IDataReader<T>, IDataTransformer<TIn, TOut>, IDataWriter<T>) for the MVFC.DataX ETL/ELT suite.
This package is part of the MVFC.DataX suite. For the full documentation and more examples, please check the main repository README.
Installation
dotnet add package MVFC.DataX.Core
Available Classes
| Class | Type | Description |
|---|---|---|
IDataReader<T> |
Interface | Reading contract — ReadAsync() returns IAsyncEnumerable<T> |
IDataTransformer<TIn, TOut> |
Interface | Transformation contract — TransformAsync() returns IAsyncEnumerable<DataResult<TOut>> |
IDataWriter<T> |
Interface | Writing contract — WriteAsync() and WriteBatchAsync() |
DataResult<T> |
Record | Result wrapper — IsSuccess, Value, Errors. Has factories DataResult.Success<T>() and DataResult.Failure<T>() |
DataError |
Record | Data error representing failures: PropertyName, ErrorMessage, AttemptedValue |
PipelineStatistics |
Record | Pipeline run stats: TotalRead, Succeeded, Failed, Skipped, Elapsed, Errors |
EnumerableReader<T> |
Class | Wraps an IEnumerable<T> or IAsyncEnumerable<T> into an IDataReader<T> |
ChannelDataReader<T> |
Class | Reader that consumes from a ChannelReader<T> |
InMemoryWriter<T> |
Class | Writer that accumulates items in a ConcurrentBag<T> accessible via the Items property |
DelegateWriter<T> |
Class | Writer that delegates batch writes to a Func<IReadOnlyList<T>, CancellationToken, Task> |
MapTransformer<TIn, TOut> |
Class | Applies a mapping function with automatic exception handling to DataResult.Failure |
FilterTransformer<T> |
Class | Filters data using a predicate. Exceptional rows become DataResult.Failure |
BatchTransformer<T> |
Class | Groups inputs into IReadOnlyList<T> of a fixed size |
SkipTransformer<T> / TakeTransformer<T> |
Class | Pagination transformers |
DistinctTransformer<T> |
Class | Removes duplicates using an optional IEqualityComparer<T> |
OrderByTransformer<T, TKey> |
Class | Sorts the stream by a key |
FlatMapTransformer<TIn, TOut> |
Class | 1-to-N expansion (Flattens iterables) |
AggregateTransformer<TIn, TAcc> |
Class | State accumulation (fold) |
Usage / Example
While you typically use PipelineBuilder to orchestrate these classes, you can use the Core classes directly to structure data flows.
using MVFC.DataX.Core.Models;
using MVFC.DataX.Core.Readers;
using MVFC.DataX.Core.Transformers;
using MVFC.DataX.Core.Writers;
var sourceData = new[] { 1, 2, 3, 4, 5 };
// 1. Create a reader from an in-memory collection
var reader = new EnumerableReader<int>(sourceData);
// 2. Create a transformer that multiplies numbers by 2, dropping odds
var transformer = new MapTransformer<int, int>(x =>
{
if (x % 2 != 0) throw new Exception("Odds not allowed");
return x * 2;
});
// 3. Create an in-memory writer
var writer = new InMemoryWriter<int>();
// 4. Manually pipe data (Pipeline engine automates this)
await foreach (var item in reader.ReadAsync())
{
var transformedStream = transformer.TransformAsync(
new[] { item }.ToAsyncEnumerable()
);
await foreach (var result in transformedStream)
{
if (result.IsSuccess)
{
await writer.WriteAsync(result.Value);
}
else
{
Console.WriteLine($"Error: {result.Errors[0].ErrorMessage}");
}
}
}
// 5. Inspect the written results
foreach (var written in writer.Items)
{
Console.WriteLine($"Written: {written}"); // Will print: 4, 8
}
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net9.0 is compatible. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.
-
net10.0
- No dependencies.
-
net9.0
- No dependencies.
NuGet packages (12)
Showing the top 5 NuGet packages that depend on MVFC.DataX.Core:
| Package | Downloads |
|---|---|
|
MVFC.DataX.Pipeline
MVFC.DataX.Pipeline module for MVFC.DataX. |
|
|
MVFC.DataX.Providers.SQS
MVFC.DataX.Providers.SQS module for MVFC.DataX. |
|
|
MVFC.DataX.Providers.RabbitMQ
MVFC.DataX.Providers.RabbitMQ module for MVFC.DataX. |
|
|
MVFC.DataX.Providers.SqlServer
MVFC.DataX.Providers.SqlServer module for MVFC.DataX. |
|
|
MVFC.DataX.Providers.MongoDB
MVFC.DataX.Providers.MongoDB module for MVFC.DataX. |
GitHub repositories
This package is not used by any popular GitHub repositories.