Hiperspace.ISO20022.pain 1.0.4

Hiperspace.ISO20022

Hiperspace Domain Specific Database for ISO20022 messages sent via SWIFT and other Financial Institutions, and stored for historical, investigation and analysis purposes in a Hiperspace store.

All XSD Validation rules are included via the Validation property that can be applied using a Horizon filter.

NB Horizon validation rules are optional, to allow the same model to be used for durable valid storage, but also for logging purposes

The main Hiperspace.ISO20022 package aggregates all document types within the ISO20022 namespace, but for specific subject areas it is better to specific packages that only contain the elements that are needed for those document types

This initial release does not merge the duplicate definitions for common elements between subject areas

Domain Specific Database

This component provides a Domain specific database, as the structure can only be changed by amending the .hilang schemas and regenerating the model. Applications can manipulate the elements and query them using LINQ expressions (C#/F#/VB.net), using SQL (especially for Parquet export) or ODATA (BI), GPU Graph search, or Cube drilldown (using Hiperspace.DB)

The Types documentation can be found of GitHub.

Showcase

ISO20022 is a showcase for the architecture of Hiperspace:

• Elements can be serialized from XML documents to naturally compressed information that does not need to carry meta-data tags in the store
• As XML Documents are stored, they are split into constituent parts, depending on the type of element:
  • Entity : Document is stored with any one of its constituents as its key, since it is only an envelope, no value is stored
  • Entity : GroupHeader is stored by its key MsgID with other attributes stored as value
  • Entity : CustomerCreditTransferInitiation is stored with the key GrpHdr which is a reference to a GroupHeader
  • Segment : CreditTransferTransaction is a segment of CustomerCreditTransferInitiation with the additional key PmtId of type PaymentIdentification
  • Value : values can be any structure of elements with Lists and Sets. Values can contain other values and entity / segment / aspect transparently by value
• Key can be any structure that unambiguously serialized (including List but not Set) to 8 MiB of data
• Value can be any structure that can be serialized to 2 GiB of data
• Indexes are created automatically when an entity is referenced with an extent reference that is not a key
• Additional indexes can be explicitly added with the @AlternateIndex property

A sample XML document

<?xml version="1.0" encoding="UTF-8" ?>
<Document xmlns="urn:iso:std:iso:20022:tech:xsd:pain.001.001.12">
    <CstmrCdtTrfInitn>
        <GrpHdr>
            <MsgId>Message-Id</MsgId>
            <CreDtTm>2024-05-10T16:10:02.017+00:00</CreDtTm>
            <NbOfTxs>1</NbOfTxs>
            <CtrlSum>510.24</CtrlSum>
            <InitgPty>
                <Id>
                	<OrgId>
                		<Othr>
                			<Id>Client-Id</Id>
                		</Othr>
                	</OrgId>
                </Id>
            </InitgPty>
        </GrpHdr>
        <PmtInf>
            <PmtInfId>Batch-Id</PmtInfId>
            <PmtMtd>TRF</PmtMtd>
            <ReqdExctnDt>YYYY-MM-DD</ReqdExctnDt>
            <Dbtr>
                <Nm>Debtor Account Holder Name</Nm>
            </Dbtr>
            <DbtrAcct>
                <Id>
                	<Othr>
                		<Id>Debtor Account Id</Id>
                	</Othr>
                </Id>
            </DbtrAcct>
            <DbtrAgt>
                <FinInstnId>
                	<BIC>BANK BIC</BIC>
                </FinInstnId>
            </DbtrAgt>
            <CdtTrfTxInf>
                <PmtId>
                	<EndToEndId>End-to-End-Id</EndToEndId>
                </PmtId>
                <Amt>
                	<InstdAmt Ccy="USD">510.24</InstdAmt>
                </Amt>
                <CdtrAgt>
                	<FinInstnId>
                		<BIC>BANK BIC</BIC>
                	</FinInstnId>
                </CdtrAgt>
                <Cdtr>
                	<Nm>Creditor Account Holder Name</Nm>
                </Cdtr>
                <CdtrAcct>
                	<Id>
                		<Othr>
                			<Id>Creditor Account ID</Id>
                		</Othr>
                	</Id>
                </CdtrAcct>
            </CdtTrfTxInf>
        </PmtInf>
    </CstmrCdtTrfInitn>
</Document>

can be loaded to Hiperspace with

var document = xmlSerialiser.Deserialize(stringReader) as Document;
document.Should().NotBeNull();
pain.Documents.BindAll(document);

and queried using LINQ (Language Integrated Query) in C# / F# / VB.net, or remotely using Hiperspace.SQL

var query = from d in read.Documents               // documents set
            from pay in d.CstmrCdtTrfInitn.PmtInf  // implicit join
            from trans in pay.CdtTrfTxInf          // implicit join
            where d.CstmrCdtTrfInitn.GrpHdr.MsgId == "Message-Id"
            select new
            { 
                d.CstmrCdtTrfInitn.GrpHdr.CtrlSum, 
                trans.Cdtr, 
                trans.CdtrAcct, 
                pay.Dbtr,
                pay.DbtrAcct,
                trans.Amt.Value.InstdAmt.Value.Ccy,
                trans.Amt.Value.InstdAmt.Value.Value
            };

// or directly              
var alter = from tran in read.CreditTransferTransaction61s
            where tran.owner.owner.GrpHdr.MsgId == "Message-Id"
            select new
            {
                tran.owner.owner.GrpHdr.CtrlSum,
                tran.Cdtr,
                tran.CdtrAcct,
                tran.owner.Dbtr,
                tran.owner.DbtrAcct,
                tran.Amt.Value.InstdAmt.Value.Ccy,
                tran.Amt.Value.InstdAmt.Value.Value
            };

The equvilent SQL for Parquet export using Hiperspace.SQL directly or via Hiperspace.DB is

SELECT tran.owner.owner.GrpHdr.CtrlSum
     , tran.Cdtr
     , tran.CdtrAcct
     , tran.owner.Dbtr
     , tran.owner.DbtrAcct
     , tran.Amt.Value.InstdAmt.Value.Ccy
     , tran.Amt.Value.InstdAmt.Value.Value
FROM   CreditTransferTransaction61s AS tran
WHERE  tran.owner.owner.GrpHdr.MsgId = 'Message-Id';

The second query is preferable since it is a single key/value lookup in Hiperspace, but the first is better if XML/JSON is needed

Hiperspace LINQ and SQL automatically insert null coalescing logic so that

• A query of a document containing a CustomerPaymentReversal will not fail if the document does not have a CstmrCdtTrfInitn.GrpHdr.MsgId
• a query of trans.Amt.Value.InstdAmt.Value.Ccy will not fail if a different kind of payment is included

Hiperspace.ISO20022 is a domain specific database

View

While the queries are intuietive for for application integration, for Business intelligence integration it is helpful to define views that can be queried directly using ODATA.

Views in Hiperspace are projections from Entity, segment (many, inheriting owner's key) and aspect (optinal, inheriting same key as owner) - predicate conditions agaist a view are translated to predicates conditions for the underlying elements, to use key/index access.

@XmlNameSpace ("urn:iso:std:iso:20022:tech:xsd:pain.001.001.12"), ValidationCheck
segment ISO20022.Pain001001.CreditTransferTransaction61
    = ISO20022.Cube.CreditTransferTransaction
        ( Dbtr          = OwnerDbtr(owner)
        , UltmtDbtr     = OwnerUltmtDbtr(owner)
        , Cdtr          = Cdtr.Id
        , ChrgsAcct     = OwnerChrgsAcct(owner)
        , CdtrAcct      = CdtrAcct
        , DbtrAcct      = OwnerDbtrAcct(owner)
        , XchgRate      = XchgRateInf.XchgRate
        , EqvtAmt_Ccy   = Amt.EqvtAmt.Amt.Ccy
        , EqvtAmt       = Amt.EqvtAmt.Amt.Value
        , InstdAmt_Ccy  = Amt.InstdAmt.Ccy
        , InstdAmt      = Amt.InstdAmt.Value
        )

The view is defined in Cube.hilang, and can be viewed directly, or using Cube Analytics:

• ISO20022.Cube.CreditTransferTransaction is declared with @CubeFact
• {InstdAmt_Close, InstdAmt_Spot, EqvtAmt_Close EqvtAmt_Spot} are @CubeMeasure(Aggregate.Sum)
• Entities {Party_Dbtr, Party_Cdtr, Party_UltmtCdtr, CashAccount_ChrgsAcct, CashAccount_CdtrAcct, CashAccount_DbtrAcct} are @CubeDimension

When combined with Hiperspace.DB streaming aggregation it is simple to retrieve the aggregate of all Transfers for a Legal Entity, Account or drilldown from top to bottom.

• Cube.md show the structure of the view
• Cube-cube.md shows the cube and fact history
• Static-physical.md shows the CreditTransferTransaction_Cube property for each @Dimension
• Cube-physical.md shows the drill

Graph

In Hiperspace information is stored as elements (relations), much like a table but with rich structure and flexible keys; Graph is a view of information projected as

• Node, with type (properties accessible from underlying element)
• Edge, with To and From properties for the Node at each end.

Party (in role) and CashAccount (in role) inherit Node/Edge views from their base elements, so CreditTransferTransaction_Cube is also a Node with Edges linking to and from each dimension, with drilldown Edges to Cube relations and rollup Edges to higher-level cubes

While almost any Element can provide a Node view literal values for TypeName ensure that only specific elements are queried:

// query only ISO20022.Static.Party
from node in ISO20022.Nodes where node.TypeName == "ISO20022.Party" select node;
// query a specific element using its key (string encoded as SKey)
from node in ISO20022.Nodes where node.SKey == "..." select node;

Elements can provide multiple Edges, ISO20022.Static.Party provides one Node (TypeName = "ISO20022.Party") and upto for Edges

• [Party]->(Party-Bank)->[Bank]
• [Party]->(Party-LegalEntity)->[LegalEntity]
• [Bank]->(Bank-Party)->[Party]
• [LegalEntity]->(LegalEntity-Party)->[Party]

When the Graph is searched (directly or via GPU-accelerated Hiperspace.DB) hiper-edges can be found linking a Legal Entity to all its roles, and all the payments to other parties

Because Graph (Node/Edge) is a universal view, externalnode() can be used to project an edge to a node stored in a different database:

• ISO20022.Static.Bank provides edges to/from a node defined in another database
• ISO20022.Static.LegalEntity provides edges to/from a node defined in another database
• ISO20022.Static.Account provides edges to/from a node defined in another database