DataScorpio 1.0.1

DataScorpio

DataScorpio is a typed query engine for .NET APIs. It parses incoming query input, validates it against an explicit profile, and applies provider-friendly filtering, sorting, search, and paging over IQueryable<T>.

It is designed for APIs that need a simple string query model today, a structured JSON query model when the client grows, and a testing layer that can validate behavior against LINQ-to-Objects or SQLite.

Packages

dotnet add package DataScorpio

Optional testing packages:

dotnet add package DataScorpio.Testing
dotnet add package DataScorpio.Testing.Sqlite

Getting Started

Create a profile for the entity you want to query. By default, nothing is queryable until the profile exposes it.

using DataScorpio.Profiles;
using DataScorpio.Querying;

public sealed class CustomerQueryProfile : QueryProfile<Customer>
{
    public override void Configure(IQueryProfileBuilder<Customer> builder)
    {
        builder
            .AllowFilter(customer => customer.Name)
            .AllowFilter(customer => customer.Status)
            .AllowFilter(customer => customer.CreatedAt)
            .AllowSearch(customer => customer.Name)
            .AllowSearch(customer => customer.Email)
            .AllowSearch(customer => customer.Region)
            .AllowSort(customer => customer.Name)
            .AllowSort(customer => customer.CreatedAt)
            .DefaultSort(customer => customer.CreatedAt, SortDirection.Descending)
            .MaxPageSize(100);
    }
}

Register DataScorpio once. The simplest path is assembly discovery:

using DataScorpio.DependencyInjection;

services.AddDataScorpio(options =>
{
    options.FromAssemblies(typeof(CustomerQueryProfile).Assembly);
});

This discovers concrete QueryProfile<TEntity> and QueryConventionSet types with parameterless constructors from the selected assemblies.

You can still register explicitly when you want full control:

services.AddDataScorpio(profiles => profiles
    .AddConventions<AppQueryConventions>()
    .AddProfile<CustomerQueryProfile>());

Execute against any IQueryable<T>:

using DataScorpio.Execution;
using DataScorpio.Querying;

var result = processor.Execute(customers.AsQueryable(), new QueryRequest
{
    Filters = "Status==Active,Name@=*ada",
    Sorts = "-CreatedAt",
    Search = "north",
    PageNumber = 1,
    PageSize = 25
});

if (!result.IsSuccess)
    return Results.BadRequest(result.Validation);

return Results.Ok(result.Result);

IQueryProcessor returns a QueryExecutionResult<T>. Successful executions contain a QueryResult<T> with items and paging metadata.

QueryRequest

QueryRequest is the string-first request model:

public sealed class QueryRequest
{
    public string Filters { get; init; }
    public string Sorts { get; init; }
    public string Search { get; init; }
    public int? PageNumber { get; init; }
    public int? PageSize { get; init; }
}

Example query:

var request = new QueryRequest
{
    Filters = "Status==Active,CreatedAt>=2026-01-01",
    Sorts = "-CreatedAt,Name",
    Search = "ada",
    PageNumber = 1,
    PageSize = 20
};

String Filtering

DataScorpio supports a Sieve-compatible filter and sort string parser.

Common filter examples:

Status==Active
Status!=Inactive
CreatedAt>=2026-01-01
Name@=*ada
(Name|Email)@=*ada
Status==Active|Pending
DeletedAt==null

Supported operators:

Sorts use comma-separated field names. Prefix a field with - for descending order:

Name
-CreatedAt,Name

Filters And Search

Filters are field-specific predicates:

Status==Active
CreatedAt>=2026-01-01

Search is one free-text term applied across the fields marked with AllowSearch(...):

var request = new QueryRequest
{
    Search = "north"
};

Use filters when the client knows the field and operator. Use search when the client has a general term and wants "find this across the searchable text fields".

Native JSON

Use IJsonQueryDescriptorParser when your API accepts structured JSON instead of strings.

using DataScorpio.Parsing.Json;

var jsonParser = serviceProvider.GetRequiredService<IJsonQueryDescriptorParser>();

var descriptor = jsonParser.Parse("""
{
  "filters": [
    { "field": "Status", "operator": "equals", "value": "Active" },
    { "field": "DeletedAt", "operator": "equals", "value": null }
  ],
  "sorts": [
    { "field": "CreatedAt", "direction": "desc" }
  ],
  "search": {
    "term": "north",
    "fields": [ "Name", "Email", "Region" ]
  },
  "includes": [ "orders" ],
  "page": {
    "pageNumber": 1,
    "pageSize": 25
  }
}
""");

var result = processor.Execute(customers.AsQueryable(), descriptor);

JSON supports:

JSON null is represented as an explicit null value. String input uses null:

DeletedAt==null

{ "field": "DeletedAt", "operator": "equals", "value": null }

Profiles And Aliases

The default public query name is the property name from the expression:

builder
    .AllowFilter(customer => customer.Name)
    .AllowSort(customer => customer.CreatedAt);

Aliases are optional and useful when the public API name should differ from the CLR property:

builder
    .AllowFilter("customerName", customer => customer.Name)
    .AllowSort("created", customer => customer.CreatedAt);

Then clients can query:

customerName@=*ada
-created

Includes are also explicit:

builder.AllowInclude("orders", customer => customer.Orders);

The core package parses and validates include names. Provider-specific include execution should live in the application or in an adapter.

Custom Filters And Sorts

Use a custom filter when the query name is a business concept instead of a single property.

public sealed class CustomerQueryProfile : QueryProfile<Customer>
{
    public override void Configure(IQueryProfileBuilder<Customer> builder)
    {
        builder
            .AllowFilter(customer => customer.Name)
            .AllowSort(customer => customer.CreatedAt)
            .CustomFilter("InRegion", (query, value) =>
                query.Where(customer => customer.Region == Convert.ToString(value.Value)))
            .CustomFilterDescriptor("CreatedWindow", (query, filter) =>
                query.Where(customer => customer.CreatedAt >= DateTime.UtcNow.AddDays(-30)))
            .CustomSort("RecentlyCreated", (query, direction) =>
                direction == SortDirection.Descending
                    ? query.OrderByDescending(customer => customer.CreatedAt)
                    : query.OrderBy(customer => customer.CreatedAt));
    }
}

Usage:

var result = processor.Execute(customers.AsQueryable(), new QueryRequest
{
    Filters = "InRegion==South",
    Sorts = "-RecentlyCreated"
});

Custom filters and sorts receive IQueryable<T>, so they can stay provider-friendly when the LINQ you write can be translated by the underlying provider.

Reusable Conventions

When the same custom query applies to every entity that implements a base class or interface, put it in a convention set and register it once.

public interface ITenantScoped
{
    string TenantId { get; }
}

public interface ICreatedEntity
{
    DateTime CreatedAt { get; }
}

public sealed class AppQueryConventions : QueryConventionSet
{
    public override void Configure(IQueryConventionBuilder builder)
    {
        builder
            .CustomFilter<ITenantScoped>("ForTenant", value =>
                entity => entity.TenantId == Convert.ToString(value.Value))
            .CustomSort<ICreatedEntity>("RecentlyCreated", entity => entity.CreatedAt);
    }
}

With assembly discovery, convention sets are picked up automatically with profiles:

services.AddDataScorpio(options =>
{
    options.FromAssemblies(typeof(AppQueryConventions).Assembly);
});

You can also register conventions explicitly:

services.AddDataScorpio(profiles => profiles
    .AddConventions<AppQueryConventions>()
    .AddProfile<CustomerQueryProfile>()
    .AddProfile<OrderQueryProfile>());

Any registered profile whose entity implements the matching contract receives those custom query names automatically.

Results And Validation

Invalid queries are rejected before execution. Unknown fields, non-filterable fields, non-sortable fields, unknown includes, invalid page numbers, invalid page sizes, and page sizes above MaxPageSize(...) produce validation errors.

var result = processor.Execute(customers.AsQueryable(), new QueryRequest
{
    Filters = "SecretInternalField==true"
});

if (!result.IsSuccess)
{
    foreach (var error in result.Validation.Errors)
        Console.WriteLine($"{error.Code}: {error.Message}");
}

QueryResult<T> mirrors the paging names commonly used by TurtlePath responses:

ASP.NET Core Usage

DataScorpio does not require an ASP.NET Core package. Build a QueryRequest from query parameters and pass your IQueryable<T> to IQueryProcessor.

using DataScorpio.Execution;
using DataScorpio.Querying;
using Microsoft.EntityFrameworkCore;

app.MapGet("/customers", (
    HttpContext http,
    CustomerDbContext db,
    IQueryProcessor processor) =>
{
    var request = new QueryRequest
    {
        Filters = http.Request.Query["filters"],
        Sorts = http.Request.Query["sorts"],
        Search = http.Request.Query["search"],
        PageNumber = int.TryParse(http.Request.Query["pageNumber"], out var pageNumber) ? pageNumber : null,
        PageSize = int.TryParse(http.Request.Query["pageSize"], out var pageSize) ? pageSize : null
    };

    var result = processor.Execute(db.Customers.AsNoTracking(), request);

    return result.IsSuccess
        ? Results.Ok(result.Result)
        : Results.BadRequest(result.Validation);
});

Example URL:

/customers?filters=Status==Active&sorts=-CreatedAt&pageNumber=1&pageSize=25&search=ada

IQueryable Providers

The core package applies queries to IQueryable<T>. That means it can run over EF Core, another ORM that exposes IQueryable<T>, or in-memory data.

Adapters that need to keep composing after DataScorpio can apply criteria without forcing a terminal page result:

var criteria = processor.ApplyCriteria(db.Customers.AsNoTracking(), new QueryRequest
{
    Filters = "Status==Active",
    Sorts = "Name"
});

if (criteria.IsMaterialized)
{
    var items = criteria.Items;
}
else
{
    var totalRows = await criteria.Query.CountAsync();
    var page = await criteria.Query.Skip(20).Take(10).ToListAsync();
}

QueryCriteriaResult<T> makes the provider transition explicit. When IsMaterialized is false, Query still supports async enumeration. When a custom filter or another operation returns an in-memory query, IsMaterialized is true and Items should be used instead of calling EF async methods on Items.AsQueryable().

DataScorpio does not ship separate EF Core, ASP.NET Core, or DynaBee packages. The core IQueryable<T> pipeline is the integration point.

Testing

Use DataScorpio.Testing to validate filters, sorts, paging, validation errors, and profile behavior without depending on your application host.

using DataScorpio.Testing;

var host = new QueryTestHost<Customer>(new CustomerQueryProfile())
    .WithSeed(
        new Customer { Name = "Ada", Status = "Active" },
        new Customer { Name = "Grace", Status = "Inactive" });

host.Apply(filters: "Status==Active", sorts: "Name", pageNumber: 1, pageSize: 10)
    .ShouldBeSuccessful()
    .ShouldBeSortedBy(customer => customer.Name)
    .ShouldContainOnly(customer => customer.Status == "Active")
    .ShouldHavePage(pageNumber: 1, pageSize: 10, totalRows: 1);

DI-based testing:

using DataScorpio.Testing;

services.AddDataScorpioTesting(profiles =>
    profiles.AddProfile<CustomerQueryProfile>());

var dataScorpio = provider.GetRequiredService<IDataScorpioTesting<Customer>>();

await dataScorpio.SeedAsync(customers);

var result = await dataScorpio.ApplyAsync(
    filters: "Name@=*ada",
    sorts: "Name",
    pageNumber: 1,
    pageSize: 10);

result
    .ShouldBeSuccessful()
    .ShouldBeSortedBy(customer => customer.Name)
    .ShouldContainOnly(customer => customer.Name.Contains("Ada", StringComparison.OrdinalIgnoreCase));

SQLite-backed testing:

using DataScorpio.Testing.Sqlite;

services.AddDataScorpioSqliteTesting(profiles =>
    profiles.AddProfile<CustomerQueryProfile>());

var dataScorpio = provider.GetRequiredService<IDataScorpioSqliteTesting<Customer>>();

await dataScorpio.SeedAsync(customers);

var result = await dataScorpio.ApplyAsync(
    filters: "Name@=*ada",
    sorts: "Name");

result.ShouldBeSuccessful();

SQLite testing is useful when you want a more realistic query provider than LINQ-to-Objects. The entity must be suitable for EF Core SQLite mapping, such as having a key.

Available assertions:

API Surface

Core:

Registration:

Sample

Run the expanded sample:

dotnet run --project samples/DataScorpio.Samples.Basic/DataScorpio.Samples.Basic.csproj

The sample includes basic filtering, sorting, paging, search, OR filters, aliases, custom conventions, null filters, native JSON, and validation scenarios.