Pqax.Cli 0.2.2

pqax

The command line of PQAX. It checks a PQAX file against the rules of the format, shows what a PQAX file holds, and collects a footprint. It is a thin layer over the Pqax and Pqax.Collector libraries: everything it does, the libraries do too.

It is packed as a .NET tool, Pqax.Cli, with the command pqax. It needs .NET 8 or later. The package is not published yet.

Commands

A file with the extension .vpax is read as a VPAX file that holds the footprint as its part Footprint.pqax. --part <name> names another part, and also makes any file read as a VPAX file.

Exit codes

Output

check on a valid file:

sample.pqax: valid, 767 checks

check on an invalid file:

part.rowCount: commits.engine has 5 values, and the manifest states 6 rows of commits in this part. (part commits.json, table commits, field engine)
part.rowCount.array.pqax: invalid, 1 issue in 767 checks

The codes are those of the rules of the conformance kit. An issue line starts with the code, followed by a colon. The last line starts with the file name.

info:

File               sample.pqax, 4307 bytes
Format             pqax 0.2
Collector          pqax-conformance-kit 0.2
Level              2
Collected          unknown to 2026-10-05 20:55:12 UTC
Data values        no
Partition values   no
Parts              6
  summary.json       sources 2, tables 2, columns 10, partitionSpecs 1, partitionFields 1, physicalSchemas 2, physicalColumns 8, writers 3, operations 5
  partitions.json    partitions 2
  files.json         files 5
  rowGroups.json     rowGroups 13
  columnChunks.json  columnChunks 52
  commits.json       commits 6
Tables             14
  sources                   2 rows
  ...
Summary            2 tables, 10 columns, 5 files, 199005 bytes, 21100 rows

Numbers are written without separators, so that a script can read them. For a VPAX file, a line such as Embedded part Footprint.pqax, deflated gives the part and whether the package stores or deflates it.

Tables

pqax tables <file> [--part <name>] [--sort waste|total|rows|files|name] [--top <n>] [--wide] [--json]

One row per table with the derived measures of the storage views, which the Pqax library computes with StorageMeasures: the same numbers as the views. A .vpax file is read as for info.

sample.pqax: 2 tables, by waste
Table         Rows  Files     Live  Non-live  Unreferenced  Deleted  Stale  Waste  Waste share
sales       17,000      3   151 KB     737 B           0 B    239 B    0 B  976 B         0.6%
dbo.orders   4,100      2  37.7 KB       0 B           0 B      0 B    0 B    0 B         0.0%

The columns are those of the storage stack of the views: the live data, the non-live files that the log still tracks, the unreferenced files that a listing found, the bytes of the deleted rows, estimated from their share of the rows, and the stale columns; the waste is their sum, and the waste share is the waste over the live data and the waste. A file of format 0.1 has no unreferenced files, and a value that the file does not have is written n/a, never 0. Bytes are in 1024-based units, as the views write them; counts have thousands separators, whatever the culture of the machine. With --wide, a share of small files from the quartiles of fileSizes, when the footprint does not keep the files, is a lower bound and is written >=75.0%.

The JSON of each table has the names of the views: stack with live, nonLive, unreferenced, deleted, stale, metadata, footers, waste, overhead, data, total, wasteShare, overheadShare; smallFiles and tinyFiles; measures with the ratios; and maintenance with the last commit, the commits per day, and the days since the last commit. python -m pqax tables writes the same.

Diff

pqax diff <older> <newer> [--part <name>] [--sort total|waste|rows|files|name] [--top <n>] [--json]

Compares two footprints of the same lakehouse taken at different times, with StorageDiff of the Pqax library, by the rules that section 13 of the object model proposes: a table is the same table when its TableId is, and otherwise, when one of the two has none, when its source, namespace, and name are; a column when its FieldId is, and otherwise by name. A .vpax file is read as for info.

Older: diff-before.pqax, 7 tables, collected 2026-10-01T06:00:00.000Z
Newer: diff-after.pqax, 7 tables, collected 2026-10-08T06:00:00.000Z
Tables: 2 added, 2 removed, 5 kept, 4 changed; by total
Footprint: 634 KB to 383 KB, -252 KB; waste: 136 KB to 58.6 KB, -77.1 KB

Change     Table                    Total before  Total after  Total change  Waste change  Files change  Rows change  Files added  Files removed
removed    legacy                         177 KB            -       -177 KB           0 B            -2      -15,000            -              -
replaced   products                       108 KB            -       -108 KB      -29.3 KB            -3       -6,000            -              -
changed    orders                         138 KB      72.8 KB      -64.9 KB      -66.4 KB            -3            0            1              4
changed    sales                          122 KB       166 KB      +44.4 KB      +18.6 KB            +1       +3,000            2              1
added      events                              -      27.3 KB      +27.3 KB           0 B            +2       +5,000            -              -
recreated  products                            -      21.5 KB      +21.5 KB           0 B            +1       +2,000            -              -
changed    raw.staging                   9.96 KB      15.2 KB      +5.27 KB           0 B            +1         +540            1              0
unchanged  audit                         17.6 KB      17.6 KB           0 B           0 B             0            0          n/a            n/a
renamed    clients (was customers)       62.5 KB      62.5 KB           0 B           0 B             0            0          n/a            n/a

Table        Column       Change   Detail
orders       price        changed  type int to bigint; size +1,000 B
sales        discount     added    double
sales        discount     removed  double
sales        amount       renamed  was amt; size +6.84 KB
...

Change is added, removed, changed, unchanged, or renamed; a table dropped and created again with a new TableId is replaced and recreated. Files added and Files removed count the files by their path, when both footprints keep the files of the table; n/a otherwise. The second table lists the columns added, removed, renamed, or changed in type, nullability, or size. Bytes and counts are written as pqax tables writes them, with a sign for a change. python -m pqax diff writes the same text and the same JSON, apart from the line ends.

Collect

pqax collect --local <folder> --output <file.pqax> [--item <name.Kind>]... [options]
pqax collect --workspace <name or id> --item <name.Kind> [--item ...] --output <file.pqax> [identity] [options]
pqax collect --adls https://<account>.dfs.core.windows.net/<container>[/<folder>] --output <file.pqax> [identity] [options]

--local reads a folder with the layout of a OneLake workspace, <item>.<Kind>/Tables/[<schema>/]<table>, such as conformance/fixtures/lakehouse. Without --item, it collects every lakehouse, warehouse, mirrored database, and SQL database of the folder. --workspace reads OneLake after a sign-in with one of the identities below; the token stays in memory. The command runs the collector with the checkpoint reader of Pqax.Collector.ParquetNet, writes the file, and validates it.

--adls reads a container of a storage account with hierarchical namespace, Azure Data Lake Storage Gen2, after a sign-in with the same identities. The URL is the DFS endpoint of the account, the container, and optionally a folder: https://<account>.dfs.core.windows.net/<container>/<folder>. The form of the ABFS driver, abfss://<container>@<account>.dfs.core.windows.net/<folder>, is accepted too. The command finds the tables under the folder as the collector says: a folder with _delta_log is a Delta table, and a folder of Parquet files with no subfolder is a Parquet table, down to 4 levels below the folder. Point the URL at a deeper folder to reach deeper tables. The command ran against a storage account on 24 October 2026, on Delta tables and a folder of Parquet files, with a person signed in through the Azure CLI and with a service principal holding Storage Blob Data Reader.

It prints one line for each table that is done, with counts and never a name, then a last line:

fixture.pqax: 2 tables, 5 footers read, 0 issues, 17 requests, 142178 bytes read; valid

The exit code is 0 when the file is written and valid, 1 when the written file is invalid, and 2 for a wrong command line or a failure. pqax collect --help prints its usage.

Identities

The identities are the same for --workspace and --adls: they all request a token for https://storage.azure.com, which OneLake and Azure Storage both take. One identity at a time. Without an identity option, the command signs in a person with a device code, which it prints: a browser window can open where nobody sees it, so the device code is the default. The other identities run without a person, from a scheduled job, a pipeline, or a notebook.

A secret is never an option value: --client-secret, --access-token, and --client-certificate-password are refused, and the value given with them is not echoed. The -env options take the name of an environment variable, and the command reads the value from it. The value is not printed by --verbose, not written in an error, and not written to the footprint. The footprint does not record who collected: the manifest has no field for it.

--env-credential reads the variables that the Azure SDK reads for a service principal:

An access token is used as it is, until it expires. When it is a JWT, the command reads its exp and aud claims, refuses a token that has expired, and warns when the audience is not https://storage.azure.com. A collection that lasts longer than the token fails with a 401.

Permissions of a service principal

A service principal or a managed identity reads OneLake when two settings allow it:

The command turns a refused sign-in or a refused request into one line with the code and the likely cause:

Permissions on a storage account

Any identity, a person included, reads a container of a storage account with a data role of Azure RBAC:

A shared key or a SAS token is not accepted: the command signs in with Microsoft Entra ID only.

Use from code

Pqax.Cli.Program.Run(args, output, error) runs a command with the given writers and returns the exit code, so that a test or a host can run it in process. CollectCommand.Run(args, output, error, host) also takes a CollectHost: the environment variables, a credential to use instead of the one of the identity options, and the HTTP client and endpoint of OneLake. --adls takes its endpoint from its URL and its HTTP client from the host. TokenCache is the token delegate of the collect command: it keeps a token of a TokenCredential until five minutes before it expires.

Pack

dotnet pack src/dotnet/Pqax.Cli -c Release -o artifacts
dotnet tool install --global --add-source artifacts Pqax.Cli