bertt.geoparquet 1.0.1

dotnet add package bertt.geoparquet --version 1.0.1                
NuGet\Install-Package bertt.geoparquet -Version 1.0.1                
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="bertt.geoparquet" Version="1.0.1" />                
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add bertt.geoparquet --version 1.0.1                
#r "nuget: bertt.geoparquet, 1.0.1"                
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install bertt.geoparquet as a Cake Addin
#addin nuget:?package=bertt.geoparquet&version=1.0.1

// Install bertt.geoparquet as a Cake Tool
#tool nuget:?package=bertt.geoparquet&version=1.0.1                

GeoParquet

Build statusNuget

.NET 8 Reader/Writer library for GeoParquet files.

https://geoparquet.org/

Specification: https://github.com/opengeospatial/geoparquet/blob/main/format-specs/geoparquet.md

Blog: https://bertt.wordpress.com/2022/12/20/geoparquet-geospatial-vector-data-using-apache-parquet/

NuGet: https://www.nuget.org/packages/bertt.geoparquet/

Sample GeoParquet JSON metadata:

{
   "version":"1.0.0",
   "primary_column":"geometry",
   "columns":{
      "geometry":{
         "encoding":"WKB",
         "geometry_types":[
            "MultiPolygon"
         ],
         "bbox":[
            3.358378252510583,
            50.750367484598314,
            7.227498450845831,
            53.55501451790761
         ]
      }
   }
}

Architecture

In this package there are functions for handling Geometadata:

1] Reading

For reading GeoParquet files there is a ParquetFileReader extension method GetGeoMetadata() to obtain the Geo metadata

2] Writing

For writing GeoParquet files there is GeoMetadata.GetGeoMetadata(GeoColumn geoColumn) static function to get the geo metadata dictionary. This dictionary can be passed to the ParquetFileWriter constructor.

See sample code below for reading/writing samples.

geoparquet-tools

In development: geoparquet-tools, a command line tool for inspecting GeoParquet files.

See releases for first version, build for Linux, Mac, Windows

Install

$ wget https://github.com/bertt/geoparquet/releases/download/0.1/geoparquet-tools-linux-x64.tar.gz
$ tar -xf geoparquet-tools-linux-x64.tar.gz
$ ./geoparquet-tools

Options:

-i <file> : inspect GeoParquet file
-h : help

Sample code

In these samples NetTopologySuite (https://github.com/NetTopologySuite/NetTopologySuite) is used for handling geometries, but any library that can handle WKB geometries can be used.

Reading

Use extension method 'parquetReader.GetGeoMetadata()':

var file = "testfixtures/gemeenten2016_1.0.parquet";
var file1 = new ParquetFileReader(file);

var geoParquet = file1.GetGeoMetadata();
Assert.That(geoParquet.Version == "1.0.0");

var rowGroupReader = file1.RowGroup(0);
var gemName = rowGroupReader.Column(33).LogicalReader<String>().First();
Assert.IsTrue(gemName == "Appingedam");

var geometryWkb = rowGroupReader.Column(17).LogicalReader<byte[]>().First();
var wkbReader = new WKBReader();
var multiPolygon = (MultiPolygon)wkbReader.Read(geometryWkb);
Assert.That(multiPolygon.Coordinates.Count() == 165);
var firstCoordinate = multiPolygon.Coordinates.First();
Assert.That(firstCoordinate.CoordinateValue.X == 6.8319922331647964);
Assert.That(firstCoordinate.CoordinateValue.Y == 53.327288101088072);

Sample reading GeoParquet file with GeoArrow encoding:

var geomColumnId = GetColumnId(rowGroupReader, "xy");
Assert.That(geoParquet.Columns.First().Value.Encoding == "geoarrow.multipolygon");

if (geomColumnId != null)
{
    var geometryArrow = rowGroupReader.Column((int)geomColumnId).LogicalReader<Double?[][][][]>().First();
    Assert.That(geometryArrow.Length == 1);
    Assert.That(geometryArrow[0].Length == 1);
    Assert.That(geometryArrow[0][0].Length == 165); //165 vertices
    Assert.That(geometryArrow[0][0][0].Length == 2); //2 points
    Assert.That(geometryArrow[0][0][0][0] == 6.8319922331647964); // longitude first vertice
    Assert.That(geometryArrow[0][0][0][1] == 53.327288101088072); // latitude first vertice
}

Writing

Use GeoMetadata.GetGeoMetadata to construct the ParquetFileWriter, store the geometries as WKB.

Result Parquet file can be visualized in QGIS:

image

var columns = new Column[]
{
    new Column<string>("city"),
    new Column<byte[]>("geometry")
};

var bbox = new double[] {  3.3583782525105832,
            50.750367484598314,
            7.2274984508458306,
            53.555014517907608};

var geoColumn = new GeoColumn();
geoColumn.Bbox = bbox;
geoColumn.Encoding = "WKB";
geoColumn.Geometry_types.Add("Point");
var geometadata = GeoMetadata.GetGeoMetadata(geoColumn);

var parquetFileWriter = new ParquetFileWriter(@"writing_sample.parquet", columns, keyValueMetadata: geometadata);
var rowGroup = parquetFileWriter.AppendRowGroup();

var nameWriter = rowGroup.NextColumn().LogicalWriter<String>();
nameWriter.WriteBatch(new string[] { "London", "Derby" });
        
var geom0 = new Point(5, 51);
var geom1 = new Point(5.5, 51);

var wkbWriter = new WKBWriter();
var wkbBytes = new byte[][] { wkbWriter.Write(geom0), wkbWriter.Write(geom1) };

var geometryWriter = rowGroup.NextColumn().LogicalWriter<byte[]>();
geometryWriter.WriteBatch(wkbBytes);
parquetFileWriter.Close();

Dependencies

Schema generation

GeoParquet metadata classes are generated from JSON schema using NJsonSchema.CodeGeneration.CSharp (https://github.com/RicoSuter/NJsonSchema), see console project 'geoparquet.codegen' for details.

Schema used:

https://geoparquet.org/releases/v1.0.0/schema.json

Roadmap

  • Add support for multiple geometry columns;

  • add writing Apache Arrow encoding for geometries;

  • add support for crs;

  • add (spatial) filters.

History

2023-11-26: version 1.0.1 - target .net 6 and 8

2023-11-16: version 1.0 - using schema v1.0.0, ParquetSharp to 13 - to .NET 8

2023-07-13: version 0.5 - using schema v1.0.0-beta.1, ParquetSharp to 12.0.1

2023-01-01: version 0.4 - using ParquetSharp 10.0.1-beta1 instead of Parquet.Net 4

2022-12-30: version 0.3.1 - make geometry column name optional in SetGeoMetadata

2022-12-30: version 0.3 - add extension method to write geo metadata

2022-12-27: version 0.2 - add extension method to read geo metadata

2022-12-23: version 0.1 - implementing reader

Product Compatible and additional computed target framework versions.
.NET net6.0 is compatible.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
1.0.1 849 11/26/2023
1.0.0 109 11/16/2023
0.5.0 214 7/12/2023
0.4.0 547 1/1/2023
0.3.1 291 12/29/2022
0.3.0 283 12/29/2022
0.2.0 304 12/27/2022
0.1.0 306 12/23/2022

version 1.0.1