DeepSharp.Verso.Notebooks 0.4.0

<img src="https://raw.githubusercontent.com/xkqg/DeepSharp/main/assets/icon.png" width="96" align="right" alt="" />

DeepSharp — deep learning in C#

The best of three worlds: TensorFlow's way of describing a network, PyTorch's way of running it, and ML.NET's way of learning from a table.

DeepSharp is the C# layer over the engines that already exist: you describe, train and use a network in C#, and the arithmetic runs on .NET's own vector maths out of the box or on a heavier engine later — swapping between them does not change a line of your model. What it adds is everything around the engine: getting your data in, the layers, the training loop, the checkpoints and the pictures. And a table that a tree learns better than a network does not have to become a network: the same prepared data is meant for ML.NET's trainers too.

0.4.0 is where it learns. Layers, losses and optimizers; a training loop that stops once the validation rows no longer improve; networks described in Keras's words or written as code, trained on the rows a pipeline prepares and saved with that pipeline as one file; the pipeline's report measuring what they learned, and the charts drawing it. Beneath them, tensors whose gradients are worked out automatically, the data half — which proposes what each column holds and names what should not be there — and a notebook to see the data in, in Verso, in an application of your own, or in your browser from DeepSharp's own server. The roadmap says what comes next, and the changelog records what each release added.

dotnet add package DeepSharp
dotnet add package DeepSharp.Pipelines
dotnet add package DeepSharp.Learners.Networks

using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("btceur-1d.csv")                                // declared, not opened
    .Declare(schema => schema
        .Timestamp("timestamp")
        .Number("close")
        .Optional("trades", ColumnKind.Number))              // a column that may have gaps
    .OrderBy("timestamp")
    .SplitByTime("timestamp", train: 0.70, validation: 0.15, gap: 1) // test is the rest
    .Ahead("close", 1, AheadAs.Return)                       // the answer: tomorrow's return
    .FillMissing("trades", With.Mean)                        // only offered after the split
    .Normalise("close", Scale.MidRange)                      // every feature between -1 and 1
    .Normalise("trades", Scale.MidRange)
    .Drop("timestamp")
    .Report(report => report.Measure(Metric.Rmse).On(Part.Validation, Part.Test).As(Shown.Numbers))
    .Build()
    .Run();

var trained = new Sequential().Dense(8).Relu().Dense(1)     // Keras's words: the widths come from the rows
    .Compile(new Adam(), new MeanSquaredError())
    .Fit(prepared, new FitOptions(seed: 42) { Epochs = 20 });

var file = trained.ToJson();                                 // the network and its pipeline, one file

The course from raw data to a validated model is declared once as an artefact and replayed, and anything that learns from the data is fitted on the training rows alone. That is the whole idea, and PDD is where it is explained.

The network learns from the training rows and is judged by the validation rows; the test rows reach it only when the report measures it, each measure beside what predicting the training rows' average would score. trained.Predict(rows) answers rows that arrive later in the answer's own units — here, a return comes back as a price — and TrainedNetwork.FromJson(file, NetworkCatalog.BuiltIn(), StepCatalog.BuiltIn()) reads the network and its pipeline back in a program that has never seen the data, refusing the network beside any other fit of its pipeline, however alike their columns are. DeepSharp.Charts draws the loss curve, the measures and a confusion matrix, as SVG.

The steps and what they learned are one file: prepared.ToJson() writes it, and PreparedData.FromJson(text, StepCatalog.BuiltIn()) reads it back in a program that has never seen the data. The catalog is the list of verbs the reader knows — add .WithIndicators() for a file that holds indicators, which read the rows in their order and so need that order said first, with .OrderBy("timestamp").

Two programs in Samples run all of this on published data, and a test runs each as it stands:

dotnet run --project Samples/DeepSharp.Sample.Pipelines -c Release -f net10.0
dotnet run --project Samples/DeepSharp.Sample.Networks -c Release -f net10.0

The first asks the Titanic passenger list what each of its columns holds before anything is declared, has a profile say what should not be there and how each is answered, and prepares the passengers and a price series. The second trains a network on each of three datasets — whether a passenger survived, a price five days on, a day's bikes hour by hour — has each pipeline's report measure it, saves it as its one file, reads it back and serves a row, and writes its charts.

A notebook to see it in

DeepSharp.Verso.Notebooks writes the same pipeline as a Verso notebook, one block per step, each block the step's own JSON — edited as text, or field by field in Verso's properties panel. "Show the data here" on a block runs the pipeline down to it and shows the rows there, each column coloured over the training rows and every row marked with the part it lands in. A box on the grid leaves a column out — it turns black — or makes it a category, and the notebook writes the step that does it. "Choose the columns" lists every column of the source with its first values and what its cells propose it holds: tick it in, and it comes in so, or out; pick its kind, make it the answer and set the answer's own values — or tick a range, and seventy bands of a flock are taken in, or made the answer, with two ticks. A profile under a block names what should not be there, and where the answer is a change to the columns — a column that hands a model the answer left out, a fare of 0 said to stand for a gap — a box beside it makes it. What the blocks decide about their columns is saved beside the notebook, and the toolbar takes a saved file over again, listing every change before it makes one and every saved decision it cannot make. It also runs the whole pipeline and exports it as the same file the chain writes.

Install it from Verso's Extensions panel. The same package runs in Verso's VS Code extension, in the browser editor verso serve opens, and inside an application of your own, through DeepSharp.Verso.Api — the Notebook page says what each needs. Or let DeepSharp's own server show it in your browser, with nothing else to install:

dotnet tool install --global DeepSharp.Verso.Serve
deepsharp-serve titanic.verso

It listens on this computer alone and answers only the address it prints, token and all.

A C# cell in the same notebook reads what the blocks declare, as text — it is there after "Show the data here" or the toolbar's run, and taken back whenever the blocks may no longer make it:

#r "nuget: DeepSharp.Pipelines.Indicators"
using DeepSharp.Pipelines;

if (Variables.TryGet<string>("deepsharp.pipeline", out var text))
{
    var folder = Variables.TryGet<string>("deepsharp.folder", out var saved) ? SourceFolder.Of(saved) : SourceFolder.WorkingDirectory;
    var declaration = PipelineDeclaration.FromJson(text, StepCatalog.BuiltIn().WithIndicators());
    var prepared = new Pipeline(declaration, rows: null, folder).Run();
}

Read on

The packages

Runs on .NET 8 and .NET 10. MIT — see LICENSE. The page deepsharp-serve serves carries Mermaid and KaTeX, each under its own MIT licence, and DOMPurify under the Apache License 2.0, named in the tool's third-party notices.