DynamicWhere.ex 3.1.0

• net6.0

  • Microsoft.EntityFrameworkCore (>= 6.0.22)
  • Microsoft.Extensions.Configuration.Abstractions (>= 6.0.0)
  • Microsoft.Extensions.Configuration.Binder (>= 6.0.0)
  • Microsoft.Extensions.DependencyInjection.Abstractions (>= 6.0.0)
  • System.Linq.Dynamic.Core (>= 1.6.7)

v3.1.0 — Date comparisons that work on every date member, segments combined in the database, five new caps, a stable code where a sentence used to be, preparation enforced whether or not a store is configured, a policy bypass through members named Root, It or Parent closed, a long In list that ended the process fixed, the names the expression parser keeps refused by name, a strict tier that no longer discloses its trace or which fields exist, forced predicates that can let rows with no value through, a declared default order for guarded queries, and refused queries written to the audit.

Security fix: a member named Root, It or Parent was read as a System.Linq.Dynamic.Core keyword. Root.Name and It.Name filtered, sorted, grouped, aggregated and projected the row's own Name, Parent threw, and an alias named root, it or parent failed in Having and Summary orders. Under ApplyPolicy the gate decided on the path the caller named while the query read the row's own column: a dynamic projection of Root.Name returned a [DwDenied] Name, a filter on it tested the denied column, and a [DwForceWhere] scope reached through such a navigation filtered the row's own column. Every expression is now parsed with a library-owned ParsingConfig with the context keywords off. ParsingConfig.Default is no longer read, so a host's changes to it no longer reach DynamicWhere queries.

Security fix: a long In list ended the process. In, NotIn, IIn and INotIn on text, and In and NotIn on Guid, number and enum members, joined their values into one flat chain, one level of expression nesting per value, and EF Core and the expression compiler walk a query tree recursively: a single condition carrying about seven hundred values overflowed the request thread's stack, guarded or not, and a stack overflow cannot be caught. A list longer than 32 values is now nested as a balanced tree of flat chains of at most 32 terms. A list of 32 or fewer is written exactly as before, so its predicate and its SQL do not change, and a longer list returns the same rows.

Behaviour change: a field path beginning with one of the expression parser's own words is refused by the library. The words are new, iif, np, isnull, is, as, cast, true, false and null, in any letter case, and only as a path's first segment: Owner.New names the member, and so do it, root and parent, whose keywords are off, and every predefined type name such as String, Math, Guid or Uri. The refusal is LogicException with the message FieldPath[{path}]StartsWithReservedName and the segment on Subject, raised where a path is validated, so conditions, orders, selects, group and aggregate fields and a DefaultOrder entry answer alike, guarded or not; under the strict tier it arrives as that clause's field denial, as every unusable name does, and the startup scan reports a DefaultOrder entry naming one. Before, seven of the words raised the parser's ParseException, true and false an InvalidOperationException, and null was read as the null literal, so the query returned no rows and no error; a typed Selects entry naming such a member worked and is now refused with the rest. Rename the property and map the column with [Column].

New: DwDates.Configure refuses a declared format whose own text ISO 8601 or a year-first date already reads, such as yyyy-MM-dd or yyyy-MM-dd'T'HH:mm:ss'Z'. Declaring one could only change what such a value means: a quoted Z is a letter, not a zone, so the format reads 12:00 as a wall time where ISO 8601 reads an instant, and on a DateTime member the ISO reading converts to the host's local time, so off UTC the two disagreed and every such value was refused as AmbiguousDateFormat on that host alone. The refusal is the same on every host, and it runs after the checks that name a sharper reason.

Fix: comparisons on a DateTimeOffset member threw. The date predicate carried a null guard whether or not the member could be null and compared every date member against a DateTime literal, so any comparison on a non-nullable DateTimeOffset threw InvalidOperationException, one on a nullable DateTimeOffset threw ParseException, and DataType.Date on any nullable date member threw ParseException. The predicate is now built from the member's own type: a null guard only where the member can be null, a literal of the member's type, .Value.Date under the guard on a nullable member, and IsNull / IsNotNull answering false / true on a non-nullable member of the entity itself. A non-nullable member reached through a navigation guards the navigation instead, so IsNull / IsNotNull and NotEqual answer by it as 3.0.0 did, forced scopes included.

Behaviour change: a date value must be ISO 8601 (a date, optionally a time, a fraction, and Z or an offset such as +03:00, +0300 or +03; a lowercase t or z, a comma before the fraction and fractions beyond seven digits are accepted too), a year-first date such as 2026/09/01, or a format the deployment declares once with DwDates.Configure(o => o.Formats.Add("dd/MM/yyyy")) or from configuration. The server's culture used to decide, so "01/09/2026" was 1 September on one server and 9 January on another. A numeric date that leads with a day or a month is now refused with the new AmbiguousDateFormat unless its order is declared, whatever its numbers — so a client finds out on its first request, not on the fifth of the month. Other forms the lenient parser accepted, such as "12:00" as today at noon, are InvalidFormat. Validation reads a date the same way the builder does. Two declared formats that read one text as different dates are refused at configuration, and so are a malformed format, one that cannot read back what it writes (hh without tt), one with no year, which the parser would complete from the clock, one with a day but no month, and two that put the day and the month in opposite orders; a single value where the list of formats belongs refuses to bind.

Change: a C# DateTime, DateTimeOffset or DateOnly placed in Values is written as year-first text instead of the month-first "09/01/2026 12:30:00", so a C# caller is never refused for an unambiguous value. A DateTime whose Kind is Local (DateTime.Now, or a value Newtonsoft.Json read from text with an offset), under DataType.DateTime on a DateTimeOffset member or a HAVING alias over one, is written with its offset, such as 2026-09-17T15:00:00+03:00, so it filters on the moment it holds: a DateTimeOffset member reads text with no zone as UTC, which would put a zoneless DateTime.Now hours away on any host outside UTC. Every other DateTime is written with no zone — under DataType.Date, so DateTime.Today compares the day it was written for; on DateTime and DateOnly members; and for a Kind of Utc or Unspecified. Text values are read as sent.

Fix: no comparison on a DateOnly member worked under either date data type (IsNull and IsNotNull did), while the policy schema told filter UIs to use DataType.Date for them. They compare as a day, against a constructor rather than DateOnly.Parse, which reads "2026-09-01" as the year 1483 on a Thai server.

Fix: a HAVING condition on a date alias gets the same predicate as a member of that type. The alias's type comes from the aggregate behind it — Minimum, Maximum, FirstOrDefault and LastOrDefault return one of the values they read — so HAVING on the latest of a DateTimeOffset column works, where it threw. A DateTimeOffset value is normalised to UTC and a value with no zone is read as UTC. DateTime members keep their previous time-zone behaviour.

Behaviour change: PageCount on an unpaged result is 1, or 0 with no rows, on filter, summary and segment results alike. It used to equal TotalCount for a filter or summary — one page per row — and to be 0 for a segment with condition sets.

Behaviour change: ToListAsync(Segment) combines its condition sets into one query the database answers. Union and Intersect join the sets' conditions, Except removes its set's rows with NOT EXISTS on the primary key, and a type with no primary key uses SQL UNION, INTERSECT and EXCEPT. The sets used to be loaded into lists and combined in memory by object reference, which was right only for a tracking query with no Selects: with AsNoTracking(), with Selects, and under ApplyPolicy, which always runs untracked, Intersect returned nothing, Except removed nothing and Union counted a row once per set. Ordering, paging, projection and TotalCount now run in the database exactly as for a filter, so only the requested page is read, Orders apply before Selects, and text sorts by the database collation rather than .NET string comparison. The provider has to translate a correlated EXISTS; a keyless type also needs every column to be comparable.

Behaviour change: ApplyPolicy(ctx) refuses a context that never went through DwPolicy.PrepareAsync with PolicyContextNotPrepared, with or without a store configured. Only a store provider used to refuse one, so an attributes-only deployment accepted the missing call and would have started refusing the day it gained a store. DwPolicyContext.IsPrepared is public; the overload taking explicit options and a resolver does not check.

Behaviour change: two new caps refuse guarded requests 3.0.0 ran. DwCaps.MaxConditionDepth (default 10) bounds how deeply condition groups nest: the top group counts as one, each level of SubConditionGroups adds one, and the caller's groups are measured before forced predicates are injected. DwCaps.MaxConditionSets (default 10) bounds how many condition sets one Segment sends, empty sets included; every set adds a condition or a subquery to the one statement a segment becomes, and a set with no conditions passes every other cap. A guarded request nested eleven levels deep, or a segment with eleven or more sets, is refused in both tiers with CapExceeded, SourceOrigin "MaxConditionDepth cap (10), request had 11", unless the deployment raises the cap. Both refuse a value below 1, freeze with the posture and bind from configuration. Unguarded calls are not affected.

Behaviour change: two more caps refuse guarded requests 3.0.0 ran, and a Count now costs. DwCaps.MaxConditionValues (default 1000) bounds the values one condition carries, comparing the largest condition of the where clause, the having clause and every segment set: an In was one comparison per value for the price of one condition and one field. DwCaps.MaxAggregates (default 50) bounds the aggregates one summary computes, through the Summary terminals and the composable Group and Summary; the group floor's own count is not counted. A guarded request over either is refused in both tiers with CapExceeded, FieldPath "*" and SourceOrigin "MaxConditionValues cap (1000), request had 1001" or "MaxAggregates cap (50), request had 51", unless the deployment raises the cap; both refuse a value below 1, freeze with the posture and bind from configuration. An aggregate with no field, such as a Count, is now charged DefaultFieldCost toward MaxQueryCost, where any number of them cost nothing. Every count cap is checked before any field name is resolved, so an oversized request that also names a field that does not exist is refused with CapExceeded, where 3.0.0 answered ConditionMustHasValidFieldName. Unguarded calls are not affected.

Behaviour change: under DwTier.Strict a guarded result no longer carries the policy trace. FilterResult<T>.Policy, SummaryResult.Policy and SegmentResult<T>.Policy carried it in both tiers, and the trace names the fields a policy dropped, the attribute or rule that sealed each one, and every injected predicate: the detail the strict tier already refused through getQueryString, sent to the caller by any API that serializes its result. New DwPolicyOptions.IncludeTraceInResult (bool?, default null) follows the tier, off under Strict and on under Convenience, and true or false overrides either. PolicyQueryable<T>.LastTrace still holds the trace. The option freezes with the posture and binds from configuration.

Behaviour change: under DwTier.Strict an unknown field and a denied field answer alike. A name matching nothing on the type was refused with LogicException ConditionMustHasValidFieldName, and a denied field with a PolicyException naming its path and the attribute or rule that sealed it, so a caller could list the columns they may not see one guess at a time. Outside a dry run an unknown name is now gated as a field denied for every feature, after the caps, and gets the refusal a [DwDenied] field gets in that clause: FieldDeniedForWhere, FieldDeniedForSelect, FieldDeniedForOrder, FieldDeniedForGroup, FieldDeniedForAggregate, or FieldDeniedForSegment anywhere in a segment. Every refusal with one of those six codes carries FieldPath "*", no RuleId and no SourceOrigin, whatever the field, and a CapExceeded refusal names no path. The trace keeps the real path and records an unknown name as Denied, with a reason that says it names nothing on the type. The same tier closes the other ways to tell them apart: inside a Segment every field refusal is FieldDeniedForSegment, whatever clause refused it; a name padded with dots or blank segments is normalized as a real path is; MaxQueryCost is checked after every field gate, so a [DwCost] weight cannot set a hidden field apart from a missing one; and MissingContextValue carries FieldPath "*" and no SourceOrigin, naming neither the scope's column nor its context key. The convenience tier, which checks the cost budget before gating, and dry runs are unchanged.

Behaviour change: ErrorCode.SelectTypeMustHaveParameterlessConstructor replaces the sentence "Select projection requires a parameterless constructor on type 'X'.", so every validation message but one is a fixed code. LogicException gains Subject, which carries the type name, and a constructor that sets it. A date refusal carries the field there, named as the caller wrote it, so an alias under ApplyPolicy is not replaced by the member behind it.

New: DwCaps.DefaultPageSize (default 0, off) gives a guarded query that sends no page a page of that size, bounded by MaxPageSize. MaxPageSize only ever bounded a caller who had already asked for a page, so the request with none returned every row. The composable Filter, FilterDynamic and Summary return the query already paged; Where, Order, Select and Group take no page and are never given one.

New: [DwForceWhere(AllowNull = true)] injects (field op value OR field IS NULL) in a group of its own, joined by And to the caller's group and to the other forced predicates, so a caller's Or cannot merge with it. It is the scope for a record that belongs to one tenant or to none, which forced predicates joined by And could not express. The context value is still required, and the widened term does not satisfy [DwRequireWhere] on the same member. Combined with IsNull or IsNotNull, or on a member that can never be null, the attribute is refused with ArgumentException. ForcedPredicate.FromConstant and FromContext refuse allowNull on IsNull or IsNotNull too, and a stored rule is refused for "allowNull": true on a null check whether or not it carries a value: a null check ignores a constant, so a widened IsNotNull would inject (field IS NOT NULL OR field IS NULL) and scope nothing. ForcedPredicate gains AllowNull and FromConstant and FromContext overloads taking it; a stored rule carries "allowNull": true in its forced object, written only when true, and a value there other than true, false or null is refused. PolicyModelValidator now reports every malformed [DwForceWhere] at startup, where it used to surface on the first query.

New: [DwEntity(DefaultOrder = "CreatedAt desc, Id")] is the order a guarded query takes when its caller sends none, through ToList, ToListAsync, ToListDynamic and ToListAsyncDynamic with a Filter, ToListAsync with a Segment, the composable Filter and FilterDynamic, and the composable Page on a source nothing has ordered or projected. The caller's own orders win and are never extended, an IQueryable already ordered keeps that order — by an OrderBy before ApplyPolicy, or by a composed Order even when the policy dropped all of its orders — a projected query, through a Select before ApplyPolicy or the guarded Select, keeps its own, and a Summary is never given one. An entry naming a field the type does not have, one that is not a field and a direction, or one the core refuses to order by, such as a collection of entities, is skipped, and a field this caller may not order by, or in a Segment may not use in a segment, is left out and recorded in the trace, never refused. A field the default keeps that is audited for Order is recorded as a use, as a caller's own order is; a field left out is not. Unguarded calls ignore the attribute and behave as in 3.0. PolicyModelValidator reports an unreadable entry, a field no query can order by and a field the type's own attributes seal against ordering as errors; an unknown field, a field only overridable attributes deny for ordering, and a field denied for segments are warnings. A [DwEntity] on a derived type replaces its base type's, as .NET attribute inheritance does, so repeat DefaultOrder and RequirePolicy there.

New: DwPolicyOptions.AuditRefusals (default false) writes every PolicyException a guarded entry point raises, and ApplyPolicy's refusal of an unprepared context, to the caller's audit buffer, drained to IDwAuditSink like a [DwAudit] event, so a caller probing for columns leaves a record. DwAuditEvent gains ErrorCode, null for a use of an audited field, and a constructor that takes it; the nine-argument constructor is unchanged. A refusal event names the field the refusal was about by its canonical path, an alias's included, and under the strict tier too, where the caller's refusal said "*"; the recorded path is cut to 256 characters, and its control, format, line separator and paragraph separator characters are escaped, so a name the caller invented cannot break or reorder a line in a log. Each refusal is recorded at most once and is never changed or swallowed, and a full buffer records nothing. Off by default because it changes what reaches a sink.

Fix: StorePolicyProvider renews MaxSnapshotAge on a poll that confirms the version it is serving. A healthy store nobody wrote to refused every guarded query one ceiling after its last write. A poll whose read was overtaken by a failed refresh reloads instead of confirming, so a stale answer cannot lift a FailClosed refusal.

Fix: the composable PolicyQueryable.Group applies the k-anonymity floor. It went straight to the engine past the summary pipeline, returning the small groups ToList(Summary) suppressed; it and the composable Summary also handed back the floor's own count column, which they no longer do.

Fix: a forced null check built from a context key failed every guarded query on its type. The key was still required, and its value landed on a null check that validation refuses. ForcedPredicate.FromContext now refuses IsNull and IsNotNull and points to FromNullCheck, and a stored rule of that shape is refused when it is read, as [DwForceWhere] already refused a ContextValue on a null check.

Fix: the reflection cache kept an access record for every field path that failed validation. Tracking ran before the path was validated, and a failed path adds no entry for eviction to remove, so under LRU, the default, or LFU every invented name a caller sent stayed recorded for the life of the process, and unique names grew memory without limit — fastest under the strict tier, which resolves every unknown name of a request. A path is now tracked only once it has validated.

v3.0.0 — Field-level policies. A new layer that decides what each caller may filter, sort, select, group, aggregate, and see. Additive: nothing enforces until you opt in, and a project with no policy attributes and no DwPolicy.Configure call behaves exactly as 2.1.5.

Entry point: query.ApplyPolicy(ctx) returns a guarded handle. Requests are sanitized before the query is built and results are transformed after they materialize; the query engine itself is unchanged.

Access control: [DwDeny] and six named sugar attributes refuse any of Where, Select, Order, Group, Aggregate or Segment per field. [DwOperators] restricts which operators may target a field. [DwEntity(RequirePolicy = true)] makes an unguarded query on the type throw instead of returning rows.

Injection: [DwAlias] gives a field a public name, renamed back on the way out. [DwForceWhere] adds a predicate to every guarded query — a tenant boundary, a soft-delete filter, an ownership check. [DwRequireWhere] makes a filter on the field mandatory.

Transformation, applied in memory after materialization: [DwMask] with nine strategies (Full, Partial, Email, Phone, Regex, Fixed, Hash, Null, Tokenize), plus [DwMutate], [DwDefault], [DwGeneralize], [DwTruncate] and [DwFormat].

Hashing and tokenizing both keep a column groupable and joinable while hiding what is in it, and differ in where the secret lives. Hash is HMAC-SHA256 keyed by DwPolicyOptions.HashSalt, which must be at least sixteen characters — whoever holds that salt can recompute every digest the deployment has emitted. Tokenize draws a random token and writes it to DwPolicyOptions.TokenVault, so the only way back is to read the vault: a store you can lock, move and revoke separately from the data. InMemoryTokenVault ships here; durable vaults are in the Redis and Entity Framework Core packages. Neither strategy hides equality, which is what makes the column usable and is documented rather than defended.

Precedence: six levels, sealed attributes first and overridable attributes last, with dynamic user, role, tenant and global rules in between. Attributes are sealed by default, so a compile-time decision cannot be lifted by a runtime rule unless you mark it Overridable.

Dynamic rules: an optional store supplies rules at runtime without a redeploy, split into a cached broad zone and a per-request narrow zone. Ships in-memory; Redis and Entity Framework Core stores are separate packages. A context must be prepared once per request with DwPolicy.PrepareAsync(ctx), and an unprepared context is refused rather than silently falling back to attributes.

k-anonymity: aggregating a transformed field is denied by default and opted into with AllowAggregate = true. DwCaps.MinGroupSize suppresses any group smaller than k, so an aggregate cannot be read off a group of one.

IT DEFAULTS TO 5 AND IS ON. A guarded grouped summary therefore suppresses any group of fewer than five rows unless you say otherwise, which is the one behaviour in this release a reader should check before upgrading — though it can change no existing caller, because the floor applies only to a guarded query and guarded queries are new here. Write MinGroupSize = 1 to switch it off and it is off, in production, with nothing refused and nothing warned about: the setting starts unset, so "off" and "never configured" stay different instructions and IsMinGroupSizeSet tells them apart. See https://doc.dynamicwhere.com/docs/policies/security.

Schema discovery: PolicySchemaBuilder describes what one caller may do with one entity, bounded by DwCaps.SchemaDepth (2), SchemaCycleLimit (2) and MaxSchemaFields (2000) rather than by the navigation cap alone — a self-referencing entity that used to enumerate 335 fields now returns 59, and the rest is reachable a subtree at a time. The response is flat with a parent on every field and node, which is a tree in adjacency form.

Configuration: the whole posture binds from IConfiguration with AddDwPolicies(section, configure). A key nothing answers to refuses to start rather than being ignored, so a misspelt cap name fails the deployment instead of silently leaving a control off. The entity catalogue, the token vault and the service provider stay in code, because they are objects rather than values.

Also: query cost budgets with [DwCost], audit trails with [DwAudit] and IDwAuditSink, field descriptions with [DwDescribe] and [DwAllowedValues], startup model validation, a dry-run mode, and a PolicyTrace on FilterResult and SummaryResult reporting what the policy did.

Breaking changes: none to the 2.x API. FilterResult<T> and SummaryResult each gain one nullable Policy property, null when the query was not guarded.


v2.1.5 — Documentation only. Fixes the XML documentation shipped with the package, which drives IntelliSense in consuming projects: an unescaped generic argument in the Select<T> comment truncated its remarks and returns text, three ToList/ToListAsync overloads were missing the getQueryString parameter description, and CacheReporting.GetQuickHealthSummary documented a parameter it does not take. The library now builds with zero warnings. No API or behaviour changes from 2.1.4.

v2.1.4 — Security and correctness fix. Recommended for all users.

Fix: condition values are now escaped before they are embedded in the generated dynamic LINQ expression. A value containing a backslash or a double quote previously ended its string literal early — a search term ending in "\" threw System.Linq.Dynamic.Core.Exceptions.ParseException ("')' or ',' expected"), and a crafted value could close the literal and append predicate logic of its own, returning rows the filter should never have matched. Values now match literally, including "\" and '"'. Affects every Text and Enum operator.

Fix: AggregateBy.Alias must now be a plain identifier (a leading letter or underscore, then letters, digits, or underscores; non-Latin letters allowed). An alias containing a comma previously appended extra terms to the generated Select projection. Aliases carrying any other separator never parsed, so nothing that worked is rejected — malformed aliases now throw LogicException("AggregationMustHasValidAlias") at validation time instead of failing later.

No API changes from 2.1.3.

v2.1.3 — Licensing: now published under the MIT license (SPDX: MIT). Free forever for commercial and personal use, with no license acceptance required. No API or behaviour changes from 2.1.2.

v2.1.2 — Fix: ordering by a field path that crosses a collection navigation (e.g. "Tags.Value" on List<Tag>) threw "No property or field 'Value' exists in type 'List`1'". Collection segments are now aggregated to a single comparable value — Min ascending, Max descending — at any nesting depth, translated to SQL and safe over empty collections in memory.