StringTokenFormatter 8.0.0

StringTokenFormatter v8.0

This library provides token replacement for interpolated strings not known at compile time such as those retrieved from data stores (file system, database, API, config files etc) and offers support for a variety of token to value mappers.

Available on nuget.org at https://www.nuget.org/packages/StringTokenFormatter.

using StringTokenFormatter;

string interpolatedString = "Hello {FirstName} {LastName}";
var client = new {
    FirstName = "John",
    LastName = "Smith",
};
string message = interpolatedString.FormatFromObject(client);

.NET versions

.NET 6, 7, 8 and .NET Standard 2.0 with C# 10 language features

Migrating from version 6

There are major breaking changes. See the v6 migration page for details on how to upgrade from version 6 to version 7.

Features overview

As well as string extensions, there are equivalent Uri extensions and a reusable Resolver class which allows for easier sharing of custom settings.

string source = "Answer is {percent,10:P}";
var resolver = new InterpolatedStringResolver(StringTokenFormatterSettings.Default);

string actual = resolver.FromTuples(source, ("percent", 1.2));

Assert.Equal("Answer is    120.00%", actual);

Tokens with formatting and alignment can be specified in the same way as string.Format (.net docs). Alternative token syntax can be selected in the settings.

Nested tokens (like prefix.name), cascading containers and other complex token resolution setups are supported through the CompositeTokenValueContainer, see Building composite containers for the helper class.

Conditional blocks of text and loops can be controlled through block commands, for example:

string original = "start {:if,IsValid}{middle}{:ifend,IsValid} end";
var tokenValues = new { Middle = "center", IsValid = true };
string result = original.FormatFromObject(tokenValues);
Assert.Equal("start center end", result);

The Value Converter settings provide Lazy loading and function-resolved values and can be extended to perform custom conversion logic after token matching but before formatting. Any value container can return a Lazy or Func which will be resolved before formatting.

See also additional features and notes for performance optimization strategies and advanced usage.

Value containers

Using properties of an object (including an anonymous object) to resolve tokens:

string original = "start {middle} end";
var tokenValues = new { Middle = "center" };
string result = original.FormatFromObject(tokenValues);
Assert.Equal("start center end", result);

Using a dictionary of values or other implementation of IEnumerable<KeyValuePair<string, object>> to resolve tokens:

string original = "start {middle} end";
var tokenValues = new Dictionary<string, object> { { "middle", "center" } };
string result = original.FormatFromPairs(tokenValues);
Assert.Equal("start center end", result);

Using an enumerable of ValueTuples to resolve tokens:

string original = "start {middle} end";
var tokenValues = new [] { ("middle", "center") };
string result = original.FormatFromTuples(source, tokenValues);
Assert.Equal("start center end", result);

Using a single name and value to resolve tokens:

string original = "start {middle} end";
string result = original.FormatFromSingle("middle", "center");
Assert.Equal("start center end", result);

Using a function to resolve tokens:

string original = "start {middle} end";
Func<string, object> func = (token) => { return "center"; };
string result = original.FormatFromFunc("middle", func);
Assert.Equal("start center end", result);

See building composite token value containers for hierarchical or cascading containers. Also custom containers.

Note: comma (,) and colon (:) should not be used in token names to avoid confusion with alignment and format values.

Settings

All interpolating methods accept an optional StringTokenFormatterSettings parameter which is used in preference to the StringTokenFormatterSettings.Global settings.

The settings record is immutable so the with keyword is used to mutate the settings, so for example to replace the global settings, something like the following can be used:

StringTokenFormatterSettings.Global = StringTokenFormatterSettings.Global with { Syntax = CommonTokenSyntax.Round };

It should be noted that whilst overriding the global is a convenient action, it can cause side effects by other code using this library. Library implementations should not update Global. Alternately, consider creating an instance of InterpolatedStringResolver which takes the settings object in its constructor and provides the common methods for expanding from different ITokenValueContainer implementations.

Creating Settings instances

Using the Global settings as the base:

var customSettings = StringTokenFormatterSettings.Global with { Syntax = CommonTokenSyntax.Round };
var expanded = "This interpolated string uses (token) as its syntax".FormatFromSingle("token", "expanded value", customSettings);

Using the default settings as the base:

var settings1 = new StringTokenFormatterSettings { Syntax = CommonTokenSyntax.Round };
// or
var settings2 = StringTokenFormatterSettings.Default with { Syntax = CommonTokenSyntax.Round };

Initially, the Global settings are the Default settings.

Settings properties

Syntax

Takes a TokenSyntax instance and defines the syntax is used for detecting tokens. Default CommonTokenSyntax.Curly.

Build-in syntax within the CommonTokenSyntax class:

Note: Token markers are case sensitive.

FormatProvider

Is used to specify the IFormatProvider applied to token values and uses string.Format to apply formatting and alignment for example: {value,10:D4}. Default CultureInfo.CurrentUICulture.

NameComparer

The comparer used by ITokenValueContainer when performing token to value look-ups. Takes a standard StringComparer. Default StringComparer.OrdinalIgnoreCase.

BlockCommands

The collection of Block Commands to be used by the InterpolatedStringExpander. Default collection from BlockCommandFactory:

See Block Commands for more information.

TokenResolutionPolicy

Controls how token values are handled by ITokenValueContainer implementations. Default TokenResolutionPolicy.ResolveAll.

The policies are:

What happens next will depend upon what else is configured:

1. if this is a CompositeTokenValueContainer then the matching will cascade to the next container
2. if UnresolvedTokenBehavior setting is set to Throw then an exception will be raised

UnresolvedTokenBehavior

Defines what should happen if the token specified in the interpolated string cannot be matched within the ITokenValueContainer. Default UnresolvedTokenBehavior.Throw.

InvalidFormatBehavior

Defines how string.Format exceptions are handled. Default InvalidFormatBehavior.Throw.

ValueConverters

Applies to token values after matching but before formatting. Converters are attempted in order so that once one has successfully converted the value then no further conversions take place. Default collection (from TokenValueConverterFactory):

They can be useful to provide post-match functionality; a great example is a when using an object which contains a property that uses a Lazy. The token matcher resolves the token marker to property and then through the ValueConverters calls the Lazy.Value and returns the value of the Lazy for formatting.

All passed through types must be handled by a Value Converter otherwise an exception is thrown.

By design ToString is not used as a value converter because calling that on a custom object will yield the object.ToString result which is often not the intended behavior. For scenarios when the ToString has been overridden, the result of calling TokenValueConverterFactory.ToStringConverter<T> can be added to the settings list of ValueConverters so that the ToString method for that specific type will be used as a valid value conversion.

HierarchicalDelimiter

Defines the prefix for HierarchicalTokenValueContainer instances. Default . (period).

See also Token Value Container Builder.

Additional features and notes

Flow of Control

When resolving the token values within an interpolated string, the following sequence is followed:

1. The InterpolatedStringParser turns a string into an InterpolatedString
2. The InterpolatedStringExpander take the InterpolatedString and processes it. For a given token
   1. The passed ITokenValueContainer provides the value based on the token name
   2. A value conversion is then attempted based on the collection of ValueConverters in the settings
   3. If the token contains alignment or formatting details, string.Format is called with the FormatProvider from the settings

Block Commands are processed by the InterpolatedStringExpander and follow the flow of steps 2.1 and 2.2 for obtaining their relevant values from tokens.

Reusing InterpolatedString instances

The InterpolatedStringParser.Parse method is responsible for identifying tokens within the source string and returning the InterpolatedString of segments. Generating the InterpolatedString takes time but can be stored and pass multiple times to the InterpolatedStringExpander.Expand method.

An example would be an email template merge whereby the same message text is used each time but with client-specific details within the ITokenValueContainer.

See also The Resolver.

The Resolver

A helper class called InterpolatedStringResolver exists to allow the easy reuse of custom settings without overriding the global default. An IoC container could be used to store the resolver for use throughout the application. The resolver contains the standard expansion methods and is in some ways a preferred option to using the string extension methods.

The resolver provides methods for both expansion of tokens from string and parsed InterpolatedString.

Building composite token value containers

The TokenValueContainerBuilder provides methods for creating CompositeTokenValueContainer instances.

Note that matching attempts are made in the order that the containers are passed to the CompositeTokenValueContainer instance which will be the same as the order that they are added to the builder. This includes nested containers.

Nested containers are supported such that {prefix.token} first matches the prefix and then uses the associated container to match the suffix. In the example below, the prefix is Account and the suffix Id exists as a property on the account object.

var account = new {
    Id = 2,
    Name = "The second account",
};

var builder = new TokenValueContainerBuilder(StringTokenFormatterSettings.Default);
builder.AddSingle("text", "Message text");
builder.AddNestedObject("Account", account);
var combinedContainer = builder.CombinedResult();

string interpolatedString = "Ref: {Account.Id}. {text}.";
string actual = interpolatedString.FormatFromContainer(combinedContainer);

Assert.Equal("Ref: 2. Message text.", actual);

The delimiter can changed in the settings.

Deep nesting is supported but discouraged, instead opt for flatter composites by adding the nested container to the top level with a separate prefix.

Block commands

Introduced in v8, Block Commands wrap literal or token segments and provide behavior configurable by container values.

The block command names are always lowercase whilst tokens abide by the TokenResolutionPolicy (as well as ValueConverters).

Conditional block

Simple boolean conditions can be used to exclude blocks of text.

The starting command format is :if,token where the token resolves to a boolean value dictating whether to include the block. The ending command is :ifend and can optionally include the token name.

string original = "start {:if,IsValid}{middle}{:ifend,IsValid} end";
var tokenValues = new { Middle = "center", IsValid = false };
string result = original.FormatFromObject(tokenValues);
Assert.Equal("start  end", result);

Nested conditions are supported.

Loop blocks

Provides a fixed number of iterations for the nested block.

For token derived iterations {:loop,token} is used and for constant iterations {:loop:value} where value is the number of iterations required. The ending command is {:loopend}.

string original = "outside {:loop,Iterations}{innerValue}{:loopend} outside";
int callCount = 0;
var called = () => {
    callCount++;
    return callCount switch
    {
        1 => "a",
        2 => "b",                
        _ => throw new IndexOutOfRangeException("Too many calls"),
    };
};
var tokenValues = new { Iterations = 2, InnerValue = called };
string result = original.FormatFromObject(tokenValues);
Assert.Equal("outside ab outside", result);

In this example, the token value InnerValue is a Func<string> which returns a different value on each call to the function.

Nested loops are supported.

Creating a custom ITokenValueContainer

Whilst there are a number of built-in containers, it many be necessary to create a complete custom container. The container should take in the settings interface ITokenValueContainerSettings and obey NameComparer and TokenResolutionPolicy properties.

See also Token Value Container Builder.

FrozenDictionary support in .NET 8

When compiling against the .NET 8 version of the library, a method Frozen is available on DictionaryTokenValueContainer and ObjectTokenValueContainer instances which sets the backing dictionary and can provide a significant performance boost when reusing instances.

Async loading of token values

There is no plan to support async/await within the library, the reason is that the library is designed to the CPU-bound and adding in an IO-bound layer massively changes the design and considered use-cases.

The InterpolatedString returned by the InterpolatedStringParser contains an extension method Tokens which provides a unique list of tokens found within the interpolated string. These token names can be used by an async method to, for example, request the token values from a data store. The token values can be loaded into an object or IEnumerable<KeyValuePair<string, T>> and provided as a parameter to the matching TokenValueContainerFactory method. The InterpolatedString and ITokenValueContainer can then be passed to the InterpolatedStringExpander.Expand method which in turns returns the resultant string.