iTool.KnowledgeHub.Idb 4.0.15-workspace.20261105.2

iTool.KnowledgeHub

Business-neutral world knowledge primitives and AgentKernel tools.

The project intentionally fixes only the mechanics of a living world:

• WorldObservation enters the world;
• WorldIntent asks for a bounded view;
• KnowledgeContext and KnowledgeExplanation are consumer projections;
• EvolutionProposal lets an Agent submit a candidate without applying it;
• EvolutionCandidate records a data-driven proposed change;
• confidence decay retires stale links without deleting their evidence;
• conflicts can coexist, be resolved by selecting a link, and remain explainable;
• every mutation is revisioned and the reference runtime can roll back to an audited revision;
• IWorldEvolutionPolicy supplies learning behavior without adding business enums.

Predicates are opaque strings, so schema statements (for example, a relation describing a domain, range or inverse) use the same evidence-backed link model as every other world statement. No schema migration is required when a new vocabulary appears.

InMemoryWorldKnowledgeRuntime is a deterministic reference runtime for tests and simulations. Production hosts can replace it with an Orleans/DataFabric implementation without changing the agent-facing protocol.

For a repeatable learning/recall workload, WorldKnowledgeCorpusFactory.CreateReferenceCorpus() produces at least 1,000 small documents from 125 topic seeds spanning astronomy, physics, biology, earth science, mathematics, computing, language, history, geography, philosophy, art, music and other neutral world topics. DocumentKnowledgeIndex is a deliberately thin lexical projection: it stores the source text for recall, while one evidence-backed document observation is written to the same World runtime. The knowledge_learn_documents and knowledge_recall_documents AgentKernel tools use that projection without introducing a second fact model.

The synthetic corpus is only a deterministic regression fixture. For real, current information, WikimediaKnowledgeSource reads public Wikipedia summaries through the MediaWiki API and maps each page to a KnowledgeDocument with its page URL, revision id, retrieval time and reuse notice. Use FetchPagesAsync for a curated/reproducible title list, or FetchRandomAsync to sample the live encyclopedia (the default count is 1,000). Missing pages and transient HTTP/JSON failures are returned as diagnostics instead of being silently learned; only documents with non-empty extracts are admitted. A revision-aware document id keeps a later page revision as new evidence instead of overwriting the earlier snapshot.

var runtime = new InMemoryWorldKnowledgeRuntime();
var index = new DocumentKnowledgeIndex(runtime);
using var http = new HttpClient();
var source = new WikimediaKnowledgeSource(http);
var live = await source.FetchRandomAsync(new WikimediaKnowledgeSourceOptions
{
    Language = "en",
    Scope = "world",
    Count = 1_000,
    BatchSize = 20,
});
if (!live.ReachedRequestedCount)
    throw new InvalidOperationException($"Only fetched {live.ReturnedCount}/{live.RequestedCount} live pages.");

await index.LearnAsync(new KnowledgeDocumentBatch
{
    BatchId = $"wikimedia-{DateTimeOffset.UtcNow:yyyyMMdd}",
    RequestedBy = "live-source-agent",
    Documents = live.Documents,
});

Live material must retain the returned Source URL and License/SourceRevision fields and should be revalidated before publication. The source follows the public MediaWiki Action API model and does not turn web text into unverifiable synthetic facts.

To let an Agent perform this as one explicit effect, register AddWikimediaKnowledge(index, source). It exposes knowledge_fetch_wikimedia, which reports fetched/applied counts and every diagnostic; the Agent can then use the existing knowledge_recall_documents tool against the same index.

DailyWorldSimulationFactory supplies a seeded day-by-day stream for integration tests and demos: people encountered, places visited, objects noticed, sounds heard, transient signals, short information notes, and occasional contradictory reports. It emits ordinary WorldObservation records plus documents, so the same stream exercises learning, forgetting, conflict repair, emergent insight and recall. Change DailyWorldSimulationOptions.Seed to replay another trace.

var runtime = new InMemoryWorldKnowledgeRuntime();
var tools = KnowledgeAgentTools.Create(runtime);

// Register `tools` with iTool.AgentKernel. The agent can observe, understand, explain, propose and evolve
// the same world; it does not carry a second memory.

// Or use the thin builder extension:
// AgentKernel.CreateBuilder().AddWorldKnowledge(runtime);

var index = new DocumentKnowledgeIndex(runtime);
await index.LearnAsync(new KnowledgeDocumentBatch
{
    BatchId = "world-reference-v1",
    Documents = WorldKnowledgeCorpusFactory.CreateReferenceCorpus().ToList(),
});
var hits = await index.RecallAsync(new DocumentRecallRequest { Query = "photosynthesis plants" });
// hits.Hits[i].Context links each document back to its world observation/evidence.

The reference behavior is covered by tests/iTool.KnowledgeHub.Tests and can be run with:

dotnet test tests/iTool.KnowledgeHub.Tests/iTool.KnowledgeHub.Tests.csproj

Production iDB RAG work (in progress)

The production target is a continuously maintained RAG engine with iDB as the authoritative document, evidence, knowledge and ingestion-state store. The in-memory reference runtime is not the production storage backend. AgentKernel remains the execution owner and MySearch remains the rebuildable lexical projection; business hosts supply identity, authorization and domain policies.

KnowledgeDocumentChunks now preserves exact UTF-16 source offsets with bounded UTF-8 chunk payloads, Unicode-safe overlap and content-addressed versions that include source metadata. iTool.KnowledgeHub.Idb/IdbKnowledgeDocumentStore stages immutable chunks and a manifest before conditionally publishing the current-document head through the existing iDB mutation contract. Readers must resolve a published head for current retrieval; direct version reads are evidence lookups and can include staged versions. A version manifest omits the body, which is reconstructed from its bounded chunk pages and offsets. Do not configure time-based iDB record expiry for these records until reference-aware maintenance is implemented: expiring evidence behind a live head would violate this contract. No production host registration is enabled yet.

Completion gates for the full production objective:

• Immutable version/chunk staging, conditional publication, evidence paging, replay and lost-ack contract tests.
• Bounded adaptive batch ingestion, resumable jobs, leases/fencing, cancellation and backpressure.
• iDB-backed world state and historical fact queries, multi-value conflict semantics, evidence lifecycle.
• MySearch projection registration, revision fences, rebuild/recovery and published-version filtering.
• Embedding lifecycle, bounded vector retrieval, hybrid fusion/reranking and explicit degraded-mode diagnostics.
• Grounded answer/context assembly with stable citations, budgets, abstention and AgentKernel integration.
• Host authorization on every operation, tenant isolation and restricted write tools.
• Rolling incremental synchronization, deletion/supersession, reference-aware retention, orphan cleanup and compaction.
• Real iDB/Redis/Orleans restart and fault-injection tests, realistic relevance evaluation, sustained ingestion/query/resource tests.

The first gate is verified with a fault-injectable iDB contract double. The adapter now batches staging by provider-reported document/byte capacity, retries bounded admission changes and verifies existing immutable rows when the provider's operation history has expired. IdbKnowledgeLexicalRetriever uses existing BM25/search-after APIs and checks authoritative heads before returning a passage; WithdrawAsync hides a document without destroying its historical evidence. SynchronizeAsync returns a fence that callers can reuse for read-your-writes queries. Search scores are nullable: sorted provider responses may supply ranking without a scalar score. Budget exhaustion is explicit.

An isolated Redis/Orleans integration method has completed all assertions for cluster recreation, version replay, lexical filtering, withdrawal and restore. Its stage trace confirms completion, but VSTest did not include the method in its results after search execution; it is recorded separately from the five passing adapter unit tests. This is not an operating-system process restart, Redis restart or sustained-load proof. The implementation uses iDB/MySearch public contracts only; changes to their implementation belong to sibling work and are outside this RAG task. All unchecked gates remain required; passing the reference corpus tests does not prove production retrieval quality or long-running resource bounds.

The test suite includes a 1,000-document learn/replay/recall check and a scripted AgentKernel run that calls the document recall tool and verifies the returned evidence, plus a scripted learn-then-recall tool chain.

The reference quality gates are intentionally explicit: the corpus must have at least 1,000 unique documents across 30 or more topic families and two or more languages; replay must apply zero new observations; a ten-query labeled benchmark at K=8 must reach Precision@K and Recall@K of at least 0.90, MRR of at least 0.90, and evidence coverage of at least 0.99; stale links must retire while their evidence remains available; dense observations must yield at least two evidence-backed, stable insight candidates and an Agent run must be able to explain those candidates from derived evidence; a recall-only AgentKernel run must complete in no more than two tool rounds/two calls with no retries or failed tool results. RecallQualityEvaluator and AgentToolRunMetrics provide the reusable measurements behind these assertions.

Rolling ingestion identity now excludes RetrievedAt (fingerprint protocol 2), while retaining source revision, content, title, scope and other source metadata. An unchanged published version only advances the head's LastRetrievedAt through iDB CAS; older observations do not move freshness backward. FirstRetrievedAt belongs to that head publication; the immutable version manifest retains the retrieval time of the first successfully staged manifest. Restoring a withdrawn version requires its current head revision. Legacy fingerprint-1 IDs remain explicitly readable; reingestion computes a fingerprint-2 ID (no destructive migration or historical evidence rewrite). This handles unchanged polling, not yet resumable source scheduling or reference-aware cleanup.

The RAG adapter now exposes IdbKnowledgeRagRetriever.RecallAsync(query, maximumUtf8Bytes, minimumSourceRevision, cancellationToken). It runs the lexical and optional vector channels in parallel, fuses ranks with RRF, rechecks iDB publication heads, and returns citation context plus explicit diagnostics. CandidateBudget applies per channel; combined examined counts are returned. A missing semantic configuration reports semantic_not_configured; classified embedding HTTP or timeout failures report embedding_service_unavailable while lexical retrieval remains available. Cancellation, invalid embeddings and storage failures propagate. Approximate semantic recall reports semantic_recall_approximate even when its candidate budget is not exhausted.

The host constructs an authorized store/collection and model profile, publishes documents, calls EmbedVersionAsync for the desired version/profile, and obtains a SynchronizeAsync fence before read-your-writes retrieval. This is explicit library orchestration; automatic durable ingestion jobs, production authorization registration and grounded answer generation are not yet implemented. An isolated real iDB integration trace verifies independent semantic recall, historical-version filtering, opposite-vector distractor exclusion, scope filtering and degraded lexical fallback with a deterministic embedding fixture. It does not establish a real model's relevance or ANN quality.

AgentKernel hosts can register the read-only production retrieval tool with builder.AddKnowledgeRag(accessResolver) from iTool.KnowledgeHub.Idb.Agents. IKnowledgeRagAccessResolver must derive the actor's identity and access from trusted host state using AgentToolContext; it returns a bound KnowledgeRagReadGrant whose Recall delegate can be IdbKnowledgeRagRetriever.RecallAsync. The grant binds scope, context bytes, per-channel candidate budget, passage limit and minimum source revision. Model input contains only Query and Limit. The handler resolves authorization on each invocation and revalidates it before returning evidence. A null grant denies access. A null grant Scope explicitly allows the entire bound base and must only be issued by a host that has authorized that access. This is an integration contract, not a built-in product identity/permission store; real product wiring remains required.

The tool serializes citations once rather than duplicating them as both JSON and typed passage content. Its description instructs consumers to treat evidence as data and abstain without support; this does not by itself implement answer validation or guarantee prompt-injection resistance.

KnowledgeAnswerGenerator makes one bounded IChatClient call using the typed citation envelope. No evidence bypasses generation and returns insufficient_evidence. The response must be structured claims with known citation labels and exact nonempty source quotes; malformed output, unknown labels or mismatched quotes are rejected as abstentions. Only verified references are returned. This checks provenance, not whether the quote entails the claim or the source is factually correct. IdbKnowledgeRagAnswerService composes hybrid retrieval with this generator and retains recall and abstention diagnostics. It requires a host-authorized binding; callers must revalidate authorization before returning the generated answer. An optional AgentKernel answer tool is available as described below.

To enable answer generation, explicitly register AddKnowledgeRag(accessResolver, includeAnswers: true) and supply the optional KnowledgeRagReadGrant.Answer delegate bound to IdbKnowledgeRagAnswerService.AnswerAsync. A read grant without that delegate cannot generate answers. The host controls MaximumOutputTokens; model arguments cannot override it. Recall and answer handlers share one grant-validation/return-time-revalidation path, including revocation during model execution. This supplies the integration boundary; real product identity wiring is still the host's responsibility.

Whitespace-only evidence chunks now persist a knowledge-embedding-skip record with reason whitespace_only. They are not passed to the embedding model and have no query buckets; subsequent runs reuse the skip record. The original evidence remains available. Embedding receipts distinguish newly Generated, Reused (including prior skips), and newly Skipped chunks.

PrepareIngestionAsync(operationId, document, expectedHeadRevision, profile) stages immutable source evidence and an immutable operation descriptor in iDB before publication or model calls. The existing host/Runtime scheduler can then deliver just the operation ID to ResumeIngestionAsync; the original document body is no longer required. Progress advances by iDB CAS through published, embedded and completed (with a search fence), or superseded. Model failure leaves published progress for retry; existing chunk vectors are reused. Completed operations do not call the model again. Old jobs do not overwrite a newer publication or resurrect a withdrawal. This is an at-least-once phase runner, not a scheduler or an exactly-once model-call guarantee. Job discovery/delivery, retry timing and single-owner execution still need host integration. Preparing source before the descriptor can leave orphan staging after a crash; reference-aware cleanup is still pending. Do not automatically delete staging based on age until that lifecycle is implemented.

DiscoverIngestionAsync queries a bounded page of RAG request records, then reads authoritative request/progress rows from iDB so stale search visibility cannot rerun completed work. Cursors bind to the knowledge base and index. RunIngestionPageAsync resolves each request's embedding profile through a host callback and executes a sequential bounded page; model availability failures return retry diagnostics without losing the persisted phase. Storage and integrity errors propagate. The existing Runtime scheduler should persist NextCursor, restart from null after end-of-scan, and schedule failed requests or another sweep with backoff. New request IDs inserted behind the current cursor are found on the next sweep. The host owns model lifetime and task delivery; this library starts no timers or autonomous processes. Requests from before discovery metadata was introduced remain resumable by explicit operation ID but need migration before indexed discovery.

Saved-answer hosts can call RetainEvidenceAsync(ownerId, versionId) before exposing durable citations, using a stable owner ID per saved artifact. References live in the same iDB knowledge base and remain active after document withdrawal; they have no implicit expiry. Repeating retain recovers a lost acknowledgement without adding another reference. ReleaseEvidenceAsync requires the reference revision and is terminal for that owner/version pair, so delayed retains cannot resurrect a released reference. A newly saved artifact uses a new owner ID. Hosts must authenticate owners and release only after the artifact no longer needs the source. Retirement must atomically fence new references/publications and account for unfinished ingestion, not merely count a stale index view.

TryRetireVersionAsync freezes an unused version without deleting its evidence. Current publication, active saved-answer references and unfinished ingestion block retirement. The bounded indexed scan uses a synchronized source revision, and retirement freezes the version guard atomically with the permanent retirement marker. Same-version writes invalidate that guard check; legacy data without a guard retains the collection-level source fence. Retired evidence IDs cannot be reused; returning content can receive a new storage identity using the current head revision, as described below. Use only after migrating prototype records that lack Kind/KnowledgeBase metadata. Contract tests cover a reference created after the scan; independent-client process probes also cover concurrent ingestion, references, retirement and collection against one shared storage host.

CollectRetiredVersionAsync performs one bounded pass (1–64 records) over an explicitly retired version. New retirement records retain an indexed fence and persist collection checkpoints; pages read authoritative rows and conditionally delete chunks and all embedding profiles without repeating synchronization. Legacy retirement records use the synchronized path. An exhausted fenced scan permits deleting the manifest. Call repeatedly until Completed; after interruption resume from the persisted checkpoint. Permanent retirement markers prevent late writers recreating old evidence IDs. Ingestion and reference receipts remain for replay, with optional request/progress consolidation; terminal ingestion no longer requires the removed manifest. This reclaims source bodies, not all lifecycle metadata. Hosts own the retention policy and pass scheduling; no automatic deletion or timer starts upon registration.

Staging now writes a knowledge-staging intent before any chunk. DiscoverStagedVersionsAsync finds intents older than a host-supplied cutoff, including interrupted staging with no manifest. Its cursor binds the index, knowledge base and cutoff; freeze the cutoff within a sweep. First staging time is not a lease or last-use time: existing heads, references and pending ingestion still block retirement. RunRetentionPageAsync combines discovery, retirement checks and one collection page per candidate, returning blockers and counts. Hosts persist NextCursor and start another sweep after reaching the end, so partially collected versions receive further passes. Collection removes the staging intent last, leaving a durable discoverable record until body cleanup is complete. Prototype orphan data written before staging intents needs explicit migration.

Semantic buckets use angular-16x16-v2: sixteen stored keys and 272 one-bit-neighbor query probes. The earlier eight-bit prototype over-selected candidates; bounded synthetic Recall@8 was 7.0% at 10k vectors/256 candidates versus 95.3% for V2. At 100k, V2 required 1024 candidates to reach 94.9% on that fixture; the default 256 still truncated. These are synthetic candidate measurements, not real-language quality or storage latency. See benchmarks/iTool.KnowledgeHub.RetrievalBenchmarks for reproducible gates and raw results. Re-run EmbedVersionAsync to build V2 projections before switching existing deployments; terminal ingestion does not automatically rebuild old protocols.

RebuildProjectionPageAsync backfills the selected model profile and the current bucket protocol over published heads. Each work unit embeds at most sixteen chunks; EmbedVersionPageAsync also exposes that bounded unit directly. The opaque cursor contains the document's chunk offset and binds to the knowledge base, index, profile and protocol. Save NextCursor only after receiving the result; replaying an earlier cursor reuses committed chunks. Model availability failures return retry without advancing that document; storage failures propagate so the host retries its last saved cursor. Source versions that are no longer current are skipped. SearchRevision fences the projections written by the page. This works independently of completed ingestion receipts.

SweepCompleted means this traversal ended, not that concurrent publications are all covered. Hosts should repeat sweeps and arrange new ingestion for the target profile during a migration; profile activation needs a separate coverage check and real-query evaluation. This API does not change the online profile or delete previous embeddings. For a protocol upgrade, run the new maintenance worker before switching query workers; the new binary queries only its own protocol. For a model upgrade within one protocol, old and new retrievers can coexist with separate profiles.

AuditProjectionPageAsync checks coverage without calling the model or writing records. Pass its opaque NextCursor unchanged until ready, incomplete, or source_changed. It checks current heads, target embedding presence, metadata bindings, whitespace skips, normalized vector payloads and recomputed bucket keys in batches of at most sixteen chunks per work unit. Counts accumulate across pages. Any change to the collection source revision invalidates the sweep, even a change in another knowledge base sharing that collection; restart at null after source_changed.

ready describes coverage at the returned SourceRevision only. It is not a lasting guarantee, semantic-quality verdict, authorization token, or atomic profile activation. Audit cursors are host-internal continuation state and must not be accepted from an untrusted client. A busy shared collection may need a quiet audit window. The staging-ledger invariant is required: migrate older published versions without staging records before relying on complete coverage. The active-profile switch uses the conditional activation API below, and ongoing ingestion must maintain the target.

ActivateProjectionAsync consumes the host's unmodified ready audit, bound to this knowledge base, index, profile and protocol. It conditions the active-record write on both the audited source revision and the expected previous active-record revision. Stale audits/configurations fail; replaying a lost acknowledgement recovers the matching active record. Audits and these management APIs are trusted-host operations, not external authorization tokens. Model quality approval remains the host's responsibility.

Construct IdbKnowledgeRagRetriever(store, search, profile => generator) to resolve the persisted active profile per request. It checks the active revision again after recall and retries once on a switch, then fails explicitly if switching continues. Without activation it provides lexical recall with the existing semantic-not-configured diagnostic. Existing explicit-profile constructors remain useful for evaluation and backfill verification; they do not follow the active record.

After activation, publication validates the active projection before the atomic head write, under the same source fence. The low-level workflow is Stage → Embed → Publish. Matching-profile ingestion requests generate embeddings before publication, so model failure leaves the previous head visible. A queued request using the single-generator overload still fails on a different active profile with knowledge_ingestion_profile_superseded. Activation and continued publication operate only through RAG APIs; direct writes to its internal rows bypass these invariants.

Use the model-resolver overload of ResumeIngestionAsync, or RunIngestionPageAsync, to migrate queued work across model switches. Each embedding phase resolves the current active profile (or the request's original profile before activation). The immutable request remains unchanged; progress records ProjectionProfile/ProjectionProtocol for the last completed embedding phase. Model failure retains pending work and the previous head. Replays reuse embeddings and terminal progress avoids resolving a model. A later source version still supersedes the old request using the original head precondition. A switch racing a publication may reject that attempt through the existing source/projection fence; retry the same operation to resolve the new active model.

Named host registration reuses the existing dependency-injection container:

services.AddIdbKnowledgeRag("articles", new KnowledgeRagBinding("knowledge", "articles"),
    sp => sp.GetRequiredService<IIndexedDocumentDatabaseGrain>(),
    sp => sp.GetRequiredService<IMySearchQueryGrain>(),
    (sp, profile) => sp.GetRequiredKeyedService<IEmbeddingGenerator<string, Embedding<float>>>(profile.Fingerprint()),
    sp => sp.GetRequiredService<IChatClient>());

The host registers the database grain for this index using the existing GetIndexedDocumentDatabase helper and the matching query grain/provider/partition, as well as each immutable model profile and optional chat client. Omit the chat callback for recall-only bindings. Resolve IdbKnowledgeDocumentStore, IdbKnowledgeRagRetriever or IdbKnowledgeRagAnswerService with GetRequiredKeyedService<T>("articles") inside the request/job's service scope. Duplicate keys fail before changing registrations; separate binding keys may target separate knowledge namespaces. Callbacks resolve host-owned clients; the RAG wrapper does not dispose them or create a timer.

This registration is not authorization. The host's IKnowledgeRagAccessResolver selects the allowed binding/scope from authenticated state, supplies that binding's RecallAsync/AnswerAsync delegates, and revalidates after the call. Do not let tool arguments choose a binding key. Long-lived agent objects must not capture scoped RAG services beyond their request scope. Existing Runtime jobs resolve the same store to run ingestion/rebuild/retention pages and persist their returned cursors.

The independent-process recovery probe in tests/iTool.KnowledgeHub.ProcessProbe forcibly kills a worker after persisted publication, then starts two new processes to verify recovery and terminal replay against real loopback Redis/Orleans. The recorded run recovered complete evidence and ready projection coverage; the first recovery made one embedding call and terminal replay made none. It does not establish Redis/machine-crash safety or sustained multi-worker performance.

Version-aware writes use the existing non-flushing published revision as an optimistic lower bound, then retry with the authoritative source revision returned on an iDB conflict. They still read retirement/active-projection state and condition the write; this avoids calling the database health API (which flushes pending indexing) before every mutation. Repeated conflicts use bounded jittered backoff. Exhausted contention is explicit and the host can retry the same operation. The rolling process probe records both contention and retained metadata growth; physical body collection alone is not a bound on total lifecycle-record count.

Ordinary chunk, embedding, request and reference writes now atomically update their version's staging row using its document revision. Retirement atomically freezes that same row and creates the permanent retirement marker. Unrelated versions therefore do not invalidate one another's retirement checks. Initial staging and head publication still use the collection source fence; publication must also serialize against an active-profile switch. Legacy versions lacking a staging row retain the conservative collection fence, with a retry if a guard appears mid-check.

The adapter reserves one mutation slot for the version guard and requires an advisory capacity of at least two records for these batches; it never splits a guarded atomic write. This changes the staging row from immutable intent to mutable technical coordination while preserving its first-staged timestamp. All writers must use the new guard protocol before per-version retirement is enabled: do not mix pre-guard binaries writing the same namespace during deployment. The code is not yet released; a live upgrade needs a quiesced writer handover.

New retirement records capture the indexed source fence and collection protocol. Collection stores its forward cursor in the frozen staging row, atomically with each page of conditional deletes. It queries at that retirement fence without flushing the index again on every page; stale indexed hits are safe because version writes are frozen and authoritative rows are read before deletion. On a lost acknowledgement, a fresh store resumes from the persisted cursor. Competing collectors cannot advance a cursor without the corresponding deletions. The manifest and staging row are removed only after the bounded scan is exhausted; permanent retirement and ingestion receipts stay. Legacy retirement records without this checkpoint protocol use the previous synchronized path.

ID-ordered maintenance and vector scans continue with an exclusive _id.keyword range rather than a live search_after anchor. The exact keyword field is required: the ordinary _id field also has analyzed tokens. This lets a saved cursor survive deletion of its anchor across worker restarts. BM25 recall fetches its bounded candidate identifiers, version IDs and ordinals in one query, using _source field selection. It then hydrates source chunks in batches of at most 64, without score pagination across changing readers. The host search window must accommodate the requested candidate budget (up to 4096; the current MySearch default is 10000); smaller host limits produce an explicit query error rather than silently reducing recall.

Lexical and vector recall share authoritative chunk materialization. Heads are checked before and after the batch read. Candidates replaced or withdrawn during that read are discarded; missing chunks for a still-current version raise knowledge_evidence_missing. Chunk identity and knowledge-base/version bindings are validated before returning source text. This is a point-in-time read, not an evidence lease: saved answers still need the existing explicit retain protocol.

CompactRetiredIngestionAsync(operationId) consolidates a terminal operation's original request into CompactedRequest on its progress record and atomically removes the standalone request. It requires a permanent retirement marker and a completed/superseded progress record; active versions keep their standalone requests, and pending operations are left unchanged. For legacy terminal receipts, the sweep may fill in the request's missing TerminalStage index hint even before retirement; it returns version_not_retired because no consolidation occurred. The host invokes it through existing maintenance work, or uses CompactRetiredIngestionPageAsync for bounded discovery and compaction (default eight operations per page). Persist its knowledge-base-bound cursor; restart a sweep to revisit blocked entries. Capacity limits or concurrent changes return an explicit blocker. Retrying a lost acknowledgement is safe, and terminal resume checks the progress receipt before requesting the now-optional standalone intent. The receipt and original intent remain indefinitely; no operation ID expiry or reuse is introduced. Deploy all readers with receipt-first resume support before enabling this maintenance step in a shared namespace. This saves one record per compacted operation but does not bound the number of historical operations or retirement markers.

Returning content does not reuse retired evidence IDs. PublishAsync and revision-aware PrepareIngestionAsync derive a new storage version from the original content fingerprint and the expected current head revision when that content's original version is retired. The manifest records ContentVersionId and RestoredFromHeadRevision; neither changes source metadata or the content fingerprint. Identical retries share the same incarnation, and refreshes of its current content reuse it. A later withdrawal/return has a new head revision and therefore a new storage identity. Old retirement markers remain authoritative, so stale writes and old citations cannot accidentally bind to newly stored bodies. StageAsync(document, expectedHeadRevision) supports the same preparation path; the original staging overload does not implicitly restore retired content. This is an additive manifest format, with optional fields omitted for original versions. Readers must treat VersionId as evidence identity, not assume it always equals the content fingerprint returned by KnowledgeDocumentChunks.Describe.

Terminal ingestion commits its progress receipt and request TerminalStage hint in the same conditional iDB batch. This hint is lifecycle metadata, excluded from immutable-intent comparison; the original operation parameters remain unchanged. Pending discovery excludes indexed terminal requests. Retirement excludes released references and terminal requests at its synchronized source fence, then still verifies returned candidates against source records. Legacy requests without a hint stay on the conservative verification path until a maintenance sweep repairs them. An exact-budget scan uses one final index-only probe to distinguish end-of-results from truncation; it never reads more than the configured number of candidate source records. These changes bound scan work by potentially blocking records rather than all historical releases/completions; they do not delete those historical records or make lifetime storage bounded.

RAG operation diagnostics

KnowledgeRagDiagnostics.SourceName is iTool.KnowledgeHub.Rag for both the .NET ActivitySource and Meter. Subscribe through the host's existing telemetry configuration; the library installs no exporter, service, background loop or logging sink. Traces inherit the caller's activity and cover ingestion.prepare, ingestion.resume, embedding, recall, generation and answer. This covers the public hybrid/answer entry points; direct low-level lexical/vector calls and maintenance methods do not currently have independent operation spans.

Operation success means the method returned normally, including explicit abstention or a normal business blocker; it does not certify answer quality. Generation includes the no-evidence fast path, so its call count is not a provider billing count. Retries internal to hybrid recall may produce multiple embedding spans; candidate measurements describe only the final returned recall. No question, document text, knowledge-base ID, operation ID, provider URL, model profile or exception message is attached. The only metric labels are fixed operation/outcome values, preventing per-document or per-tenant cardinality growth. Provider exceptions propagate unchanged to the host; traces record only the fixed operation_failed description. Host-supplied telemetry callbacks and exporters remain host responsibilities.

Current production acceptance boundary

The reference Utopia README describes a complete application with hybrid retrieval, evidence-linked answers, ingestion jobs and temporal knowledge governance. This foundation reuses its relevant RAG ideas through existing iDB, model and Runtime contracts; the product UI, identity policy and graph/ontology application are outside this RAG implementation.

These are separate gates. Local synthetic-model tests and short rolling probes are not a blanket production sign-off, nor evidence of a lifetime storage bound.

The main iTool.Develop.Solution.sln includes iTool.KnowledgeHub.Idb, its adapter tests, the independent-process/rolling probe and the retrieval benchmark. Redis-backed probes remain explicitly configured test runs; ordinary adapter test runs skip the real Redis integration when no isolated endpoint is provided.

RecallQualityEvaluator.EvaluatePassagesAsync reuses the existing quality contracts for live or recorded RAG results. It computes document-level precision/recall/MRR within the first K passages, deduplicates chunk hits, checks citation identity/offset structure and retains candidate-budget diagnostics. The retrieval benchmark's --evaluate-passages mode accepts labeled recorded runs and returns a nonzero exit code for a missed gate or invalid input. See its README and fixture schema. These mechanics are tested, including a real-iDB callback; no real-model relevance gate has yet been passed, and citation structure is not semantic entailment.

Both iTool.KnowledgeHub and iTool.KnowledgeHub.Idb follow iTool.SharedVersion.props; the adapter package includes this README and its XML API documentation. The package-consumer probe restores the exact adapter package into a separate project with no source project references, verifies all locally built dependency package hashes and exercises the public registration/evidence APIs. It produces only temporary local packages; it does not publish or deploy a Foundation release.