LightStudio.sqlite-vec 0.1.9.3

LightStudio.sqlite-vec

Native-only sqlite-vec 0.1.9 loadable extensions and browser-WASM static libraries. The package has no NuGet dependencies and is independent of LightStudio.Onnx. It contains vec0 libraries, our own SQLite 3.50.4 builds for all four WASM variants, licenses, build metadata, and MSBuild integration. No desktop/Android SQLite engine, SQLite source, or managed SQLite binding is shipped.

Runtimes

Shared assets live in runtimes/<rid>/native/. The .NET SDK selects and deploys them. The Android filename has the lib prefix required for APK native libraries. iOS, native 32-bit, and musl RIDs are not included. No AVX/AVX2-only CPU baseline is imposed.

SQLite Host

For desktop and Android, bring an SQLite engine with extension loading enabled and the managed bindings of your choice. Use SQLite 3.38 or newer for the complete vec0 query features. The extension is compiled against the ordinary sqlite3ext.h interface and receives SQLite functions through the sqlite3_api_routines table. It neither embeds SQLite nor imports a particular SQLite DLL, so it can be used with an application's existing compatible engine, including sqlite3 or e_sqlite3.

Windows can use the system winsqlite3.dll. No different extension build or SQLite import library is needed. Microsoft's WinSQLite 3.51.1 was checked under Wine with a diagnostic Windows x64 extension: extension loading was enabled, and vec0 scalar, bit-vector, KNN, and DELETE checks passed. This is a host-ABI compatibility check, not validation of the MSVC release binaries on native Windows. Both Windows CI runners probe their actual system DLL and also run the mandatory normal-SQLite consumer tests.

For Microsoft.Data.Sqlite applications using the Windows system engine, select Microsoft.Data.Sqlite.Core plus SQLitePCLRaw.bundle_winsqlite3 in the app, as described in Microsoft's provider documentation. Those are application dependencies, not dependencies of this package. WinSQLite versions and compile options depend on Windows servicing. If an older system engine lacks the required features or extension loading, select a normal SQLite engine in the application instead. No fallback desktop SQLite engine is added to this package, and neither winsqlite3.lib nor sqlite3.lib is linked into vec0.

Consume

dotnet add package LightStudio.sqlite-vec --version 0.1.9

For a desktop application already using Microsoft.Data.Sqlite, load the native asset with the explicit entry point sqlite3_vec_init:

using System.Runtime.InteropServices;
using Microsoft.Data.Sqlite;

string nativeName = OperatingSystem.IsWindows() ? "vec0.dll"
    : OperatingSystem.IsMacOS() ? "vec0.dylib" : "vec0.so";
string extensionPath = Path.Combine(AppContext.BaseDirectory, nativeName);
if (!File.Exists(extensionPath))
{
    extensionPath = Path.Combine(AppContext.BaseDirectory, "runtimes",
        RuntimeInformation.RuntimeIdentifier, "native", nativeName);
}

using var connection = new SqliteConnection("Data Source=:memory:");
connection.Open();
connection.LoadExtension(extensionPath, "sqlite3_vec_init");
using var command = connection.CreateCommand();
command.CommandText = "SELECT vec_version()";
Console.WriteLine(command.ExecuteScalar());

SQLite's loader does not use .NET's NuGet probing rules, so pass the resolved path, not just vec0, for RID-less desktop builds. RID-specific publish puts the selected native asset beside the executable. On Android, load libvec0.so from the app's native-library location using an engine/provider that exposes extension loading; Android's framework database API is not sufficient. Load the extension on each connection that uses it. Do not pass an SQLite connection handle from one engine to another. Only load trusted extension binaries.

Browser WASM

For RuntimeIdentifier=browser-wasm or TargetPlatformIdentifier=browser, the package adds NativeFileReference items for both libvec0.a and libsqlite3.a automatically, following the FFmpeg package's variant selection. Both archives come from the selected directory:

Use the .NET WASM native-build workload matching your SDK. CI uses Emscripten 3.1.69 for em3 and 6.0.2 for em6, as do FFmpeg and Photos. Framework-major selection is best-effort, not a guarantee for every workload/toolchain version. Rebuild with your SDK's Emscripten version if required. Enable multithreading in the consuming app with:

<PropertyGroup>
    <WasmEnableThreads>true</WasmEnableThreads>
</PropertyGroup>

The bundled SQLite engine uses SQLITE_THREADSAFE=0 for ST and SQLITE_THREADSAFE=1 with -pthread for MT. Column metadata, FTS5, and RTree are enabled; dynamic extension loading is omitted. SQLite's public-domain notice is included under licenses/browser-wasm[-mt]-em<major>/sqlite3/.

Supply managed bindings targeting this engine's sqlite3_* symbols, such as DllImport("__Internal", EntryPoint = "sqlite3_open"). Do not also link a second SQLite engine, including one supplied by a managed provider bundle. Providers targeting e_sqlite3 need configuration to use the bundled engine; this package does not supply aliases or a managed provider. The vec0 archives are built with SQLITE_CORE: they call the bundled engine directly, without embedding a second copy. Dynamic LoadExtension is not supported in the browser. Register sqlite3_vec_init through sqlite3_auto_extension before opening connections, or call sqlite3_vec_init(database, &error, NULL) for each existing connection using its actual native SQLite handle. Registration must reference the entry point so the native linker retains it. A managed native call can use DllImport("__Internal", EntryPoint = "sqlite3_vec_init"). The package links both archives but does not register vec0 automatically.

Thread-enabled browser hosting also requires cross-origin isolation (COOP/COEP headers) and SharedArrayBuffer support.

Build and Publish

Sources and the MIT license are SHA-256-pinned. The SQLite amalgamation archive provides headers for all builds and sqlite3.c for the separate WASM-only libsqlite3.a; SQLite is never compiled into vec0 itself. WASM build metadata records the SQLite version, source archive hash, features, and threading mode. Native build outputs are staged under artifacts/sqlite-vec-<rid>/; WASM outputs use artifacts/sqlite-vec-browser-wasm[-mt]-em<major>/, derived from the active emcc version. The generated source copy includes compatibility fixes: the upstream MSVC ARM64 Hamming-distance fallback accepts a 64-bit argument instead of truncating it to 32 bits, and WASM32 uses the 64-bit popcount builtin rather than the 32-bit unsigned long builtin. The downloaded upstream source remains unchanged.

bash scripts/build-sqlite-vec.sh linux-x64
bash scripts/check-sqlite-vec-native.sh linux-x64
bash scripts/test-sqlite-vec-package.sh linux-x64 0.1.9-local.1

bash scripts/build-sqlite-vec.sh browser-wasm
bash scripts/build-sqlite-vec.sh browser-wasm-mt
bash scripts/test-sqlite-vec-wasm.sh browser-wasm
bash scripts/test-sqlite-vec-wasm.sh browser-wasm-mt
bash scripts/test-sqlite-vec-package.sh browser-wasm 0.1.9-local.1

dotnet pack package/sqlite-vec/LightStudio.sqlite-vec.csproj -c Release -o artifacts/packages

Run the WASM builds and smoke tests once with Emscripten 3.1.69 activated and once with 6.0.2, then run the WASM package test after all four outputs exist. The final command requires eight shared-library outputs and all four WASM outputs. browser-wasm-mt is a build variant, not a NuGet RID; selecting browser-wasm always packs both threading modes for both SDK majors. Local subset packs require both SqliteVecAllowPartialPackage=true and a prerelease PackageVersion; select RIDs with SqliteVecRuntimeIdentifiers. For multiple RIDs, pass the semicolon-separated list as an environment property. SqliteVecArtifactsPath can select a different artifact root.

Linux releases build in scripts/sqlite-vec-linux.Dockerfile, which pins Ubuntu 24.04 and is independent of the ONNX build. Run Linux builds on the matching host architecture; use MSVC C++ tools, CMake, and Ninja on Windows, Xcode command-line tools on macOS, and Android NDK r28c for Android. Set ANDROID_NDK_HOME for Android builds. SQLITE_VEC_BUILD_DIR isolates a build tree; SQLITE_VEC_ARTIFACTS_DIR changes the staging root. Windows builds use Ninja with cl from an MSVC developer environment initialized for the requested x64 or ARM64 target before starting Bash. The workflow sets this up automatically. If a local build tree used a Visual Studio generator, select a fresh tree with SQLITE_VEC_BUILD_DIR before retrying with Ninja. For WASM, activate Emscripten so emcmake, emcc, emar, and emnm are available. The MT archive is compiled with -pthread; the ST archive is not. LLVM_NM can select the matching LLVM symbol tool when emnm is unavailable.

docker build -f scripts/sqlite-vec-linux.Dockerfile -t lightstudio-sqlite-vec-build scripts
docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp \
    -e SQLITE_VEC_BUILD_DIR=/work/artifacts/build/sqlite-vec-release-linux-x64 \
    -v "$PWD:/work" -w /work lightstudio-sqlite-vec-build \
    bash scripts/build-sqlite-vec.sh linux-x64

The consumer tests restore from a fresh local feed/cache, inspect the package and native architectures, exercise missing-input/release/platform guards, and compare RID-published native bytes against the staged library. Desktop jobs execute scalar, 64-bit Hamming, KNN, UPDATE, and long-metadata DELETE checks. Their Microsoft.Data.Sqlite reference provides a test-only SQLite engine and is never included in LightStudio.sqlite-vec. Android jobs check binaries and package contents; they do not execute a device test. WASM jobs inspect both archives for each SDK/threading variant, check MSBuild selection for browser apps and class libraries, reject missing SQLite archives and notices, and execute the SQL checks under Node with the bundled engine. The checks also verify SQLite's version, threading mode, FTS5, and RTree. The MT test executes SQL on a pthread worker. These tests do not publish or launch a .NET browser application.

The independent sqlite-vec.yml workflow builds on tags sqlite-vec-v<version> or manual dispatch. Tags build and upload only. Manual dispatch with publish=true publishes the complete package to NuGet.org using NUGET_API_KEY, after all build and validation jobs pass.