BioFSharp.INSDC.Crawler 0.3.0

.NET 8.0 This package targets .NET 8.0. The package is compatible with this framework or higher. 
dotnet add package BioFSharp.INSDC.Crawler --version 0.3.0
                    


                    

                
NuGet\Install-Package BioFSharp.INSDC.Crawler -Version 0.3.0
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package. 
<PackageReference Include="BioFSharp.INSDC.Crawler" Version="0.3.0" />
For projects that support PackageReference, copy this XML node into the project file to reference the package. 
<PackageVersion Include="BioFSharp.INSDC.Crawler" Version="0.3.0" />
                    


                            Directory.Packages.props
                        

                    

                
<PackageReference Include="BioFSharp.INSDC.Crawler" />
                    


                            Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package. 
paket add BioFSharp.INSDC.Crawler --version 0.3.0
                    


                    

                
#r "nuget: BioFSharp.INSDC.Crawler, 0.3.0"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package. 
#:package BioFSharp.INSDC.Crawler@0.3.0
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package. 
#addin nuget:?package=BioFSharp.INSDC.Crawler&version=0.3.0
                    


                            Install as a Cake Addin
                        

                    

                
#tool nuget:?package=BioFSharp.INSDC.Crawler&version=0.3.0
                    


                            Install as a Cake Tool

BioFSharp.INSDC.Crawler

Collects INSDC records and related raw artifacts from public archives.

Given a project or study accession, the crawler discovers connected studies, samples, experiments, and runs through ENA, fetches their XML records, and can persist both records and connectivity through BioFSharp.INSDC.SQLite.

It can also materialize a round-tripped INSDC XML tree, Europe PMC JATS or PMC Open Access PDF full text, and DEE2 count bundles. crawlAll composes the raw R2 layout. crawlR1Formats composes R1A and R1B while sharing discovery and non-paper downloads. R1 and R2 are crawler formats, not ArcIR F2 implementations.

Crawls are strict by default: an exhausted record batch fails after in-flight work finishes instead of silently returning a partial result. Explicit inspection workflows can opt into CrawlOptions.ContinueOnPartialFailure. Cancellation is never retried; transient failures use bounded retry delays. Fetch and FetchBytes remain injectable, so the normal suite uses committed offline fixtures.

Artifacts are written through same-directory temporary files and atomically renamed. Resume skips valid existing XML, JATS, PDF, and ZIP artifacts; malformed binary responses never replace the final path. SQLite persistence explicitly opts into the store's documented soft-reference foreign-key mode.

Because its HTTP stack requires .NET 6 or later, this package targets net8.0 instead of the netstandard2.0 baseline of the other shipped libraries. It is nevertheless a normal packed and published NuGet package.

Part of BioFSharp.INSDC. Released under the MIT license.

Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 was computed.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed.  net10.0 was computed.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
0.3.0 102 8/28/2026
0.2.1 133 7/9/2026
0.2.0 120 7/9/2026

Repository stabilization, target-neutral ArcIR extraction, immutable ArcIR state persistence, additive semantic enrichment, complete F1 accounting, and the selected-literal curation primitive.
- Added the packable, target-neutral `BioFSharp.ArcIR` `netstandard2.0` package with validated absolute IRI identities, normalized identity-keyed graph collections, shared term definitions, explicit add/upsert/merge operations, validation, and persistence contracts.
- Added canonical `.arcir.json` 1.0 persistence with a packaged JSON Schema, deterministic UTF-8/LF output, strict versioned decoding, invariant tagged values, authoritative identity keys, atomic create-new state publication, and SHA-256 artifact revisions.
- Added typed RFC 6901 fragment locations for every addressable graph entity and atomic value occurrence, including relation-property annotations, plus digest-verified resolution that keeps scalar values ID-less and provenance outside the graph state.
- Added strict inverse parsing for supported ArcIR JSON selectors and the immutable `LiteralMapping.apply` transformation for mapping one exact string occurrence to an already registered term while preserving the source literal.
- Added format-neutral additive semantic mapping and a `PolyglotSSSOM` adapter that preserves complete candidate claims, resolves only declared CURIEs, and leaves mapping selection and provenance to the integrating application.
- Added deterministic occurrence-level F1 accounting and resolvable source/output designations for all eight supported INSDC entities plus supplementary paper and count metadata.
- Removed the unused `BioFSharp` umbrella-package dependency and replaced the remaining preview dependency with stable `PolyglotSSSOM` `0.1.0`.
- Converted `BioFSharp.INSDC.ArcIR` into an INSDC-specific F1 adapter. All eight entity converters and supplementary paper/count ingestion now mint deterministic assertion/relation identities and consume the neutral core without retaining the proof-of-concept model shapes.
- Added a core-only test project and adapter regression coverage for conflict reporting, reference validation, deterministic identities, complete term resolution, and deterministic eight-entity fixture conversion.
- Limited the nested FAKE solution build to one MSBuild node, preventing runaway worker fan-out and silent process-limit failures in captured Windows builds.
- Made package creation reuse the already verified Release build outputs instead of rebuilding projects and risking locked output assemblies.
- Added dependency-vulnerability and generated-artifact drift gates to the full FAKE test path; generators now produce deterministic committed output.
- Added a versioned, forward-migrated SQLite schema with explicit foreign-key modes, transactional public writes, and exact fixture round-trip coverage for all five stored entities.
- Hardened crawler cancellation, bounded retries, strict partial-failure handling, upstream parsing, atomic writes, validation, and resume behavior while preserving injectable offline fetch seams.
- Split the test suite into focused IO, ArcIR, SQLite, crawler, and crawler-hardening modules; refreshed package-boundary and ArcIR terminology documentation; removed the stale local fsdocs publication path.
- Removed the unused structural-decompilation overlay from the proof-of-concept ArcIR package; its explicit converters remain the current mapping surface.