Yotei.Tools.DynamicParser 0.5.6

Yotei Dynamic Lambda Expression Parser

Yotei's Dynamic Parser is used to extract the logic of a DynamicLambda Expression (DLE) into a neutral and abstract format that doesn't depend on the capabilities of any underlying type, and that can be then used for a variety of purposes.

Introduction

Regular lambda expressions do not support using dynamic arguments. Indeed, if you try to do so the compiler will complain:

  void Foo(Expression<Func<dynamic,...>> func);

In some scenarios this is a limitation. For instance, we may need to write some arbitrary logic without tying it to a concrete type. A canonical example is to write database code that refer to columns for which there are no corresponding properties in any type of the application code.

We often solved this situation by creating a multiplicity of DTO classes, or if this is not convenient (or sometimes not even possible), reverting to use plain strings with that code. But we all know that this approach has many disadvantages including, for instance, the possibility of writing dangerous code, or making it difficult to parse such code or to keep it in sync with what your favorite ORM is using.

Wouldn't it be nice to write something like the following, where the Id name is not tied to any type, and then let that Where() method parse the expression into a neutral and abstract logic representation?

var result = database.Where(x => x.Id == "007");

DynamicParser to the rescue

And this is what the DynamicParser class does. Its static Parse() method takes a single argument, a Func<dynamic, object> dynamic lambda expression that takes a single dynamic argument, and emulates what otherwise the compiler should have done by parsing that expression, and then capturing the chain of dynamic operations binded against that dynamic argument.

var parser = DynamicParser.Parse(x => ...);
var argument = parser.Argument;
var result = parser.Result;

The Parse method returns an instance of DynamicParser that contains two main properties. The first one, Argument, describes the dynamic argument used in the dynamic lambda expression. The second one, Result, contains the last node of the parsed chain of dynamic operations. It is an instance of a class derived from the DynamicNode type, and each class contains the appropriate properties that permit your application to walk up that chain as needed.

What is DynamicParser used for?

DynamicParser is an infrastructure library. Its role is to provide the ability of parsing arbitrarily complex dynamic lambda expressions into a neutral and abstract logic representation (*).

This representation is based on classes that derive from the DynamicNode type, and they are essentially POCO classes that can be easily manipulated as needed.

(*): So, it does not provide the translation of that logic into any language (such as your favorite SQL dialect). If you are interested in this use case, please check the Yotei ORM family of libraries.

Examples

Let's suppose we want to filter a set of results by checking if the values of an arbitrary field start with some letter or bigger. In our scenario, we have not an application-level type that contains that field, so we cannot write a Linq query or anything like that. We can easily write this logic in C# terms as follows:

var parser = DynamicParser.Parse(x => x.LastName >= "L");
var result = parser.Result;

But... wait a moment! We are comparing something whose type is unknown against a string literal... which is something not tipically allowed by your C# compiler. The magic here is that x is a dynamic object, and so, it follows late bound rules. The compiler will delegate to the DLR (Dynamic Language Runtime) the responsibility of dealing with that expression. When parsed with DynamicParser, its operations are then captured as appropriate, without the need of any concrete type methods or members.

Often, we don't deal directly with DynamicParser instances, nor with their DynamicNode results. As said before, this is an infrastructure library that let you obtain a representation of that logic, that you can later use as needed - for instance, translating it into an appropriate SQL dialect.

DynamicParser can parse much more complex dynamic lambda expressions. Some examples are:

x => x.Id = x.Id + "_Whatever";
x => x.Indexed[x.Index + x.Beta];
x => x[7, "other"] = null!;
x => x(x.Argument);
x => x.MyMethod(x.Alpha, null, "other");
x => x(x.Alpha = x.Beta)(x.Beta = x.Alpha);
x => x.Alpha<int, string>(x.Beta);
x => x.Alpha(x.Alpha = x.Alpha(x.Beta = x.Alpha)[x.Alpha = x.Beta]);
x => x.Alpha == (x.Alpha = x.Beta);
x => x.Alpha && x.Beta;
x => x.Alpha += x.Beta;
x => x.Alpha = (string)x.Beta;

How it works

In a nutshells, by executing the dynamic lambda expression and intercepting each of the dynamic operations, translating them into the appropriate DynamicNode instances.

DynamicNode inherits from DynamicObject, which provides the ability of binding dynamic operations by overriding their associated methods. But this approach only works well for a subset of operations (for the purposes of DynamicParser). So, because DynamicObject implements the IDynamicMetaObjectProvider interface, DynamicNodeinstances override its GetMetaObject(...) method to generate ad-hoc DynamicMetaObject instances that are the ones that, ultimately, bind the most complex dynamic operations.

Now, for performance reasons, the DLR uses a sophisticated cache mechanism, which is based on the type of the call site, and on the type of its arguments. And this is unfortunate because DynamicParser needs to produce brand new results each time it parses a dynamic lambda expression, or any of its intermediate results.

So, to prevent recycling previous results, each DynamicNode is associated with an internal version number. Then DynamicParser forces the DLR to invalidate the cached entries by using an ad-hoc BindingRestrictions instance that validates the internal version and, if needed, updates it to a new one.

Easy to say, hard to achieve because there are three special cases.

The first one relates to member set operations, as in 'x => x.Id = ...' where, by default, the DLR 'forgets' the right side of the expression, losing that part of the logic. To prevent this, set operations implement a LastNode hack whose value is checked and used if appropriate.

The second one relates to convert (or cast) operations, as in 'x => (string)x.Id'. The thing here is that the DLR expects and validates receiving an actual instance of that type, not a DynamicNode one, and so it would fail throwing an exception. To deal with this scenario, DynamicParser produces an intermediate result of the appropriate type, boxed if necessary, a surrogate that is used as the key of a dictionary of converted results. Then, when that surrogate result is later used, it is substituted by the appropriate DynamicNodeConvert instance.

And, finally, the third one relates to nested indexed and invoke operations, as in 'x => x[x[x[...]]]', 'x => x(x(x(...)))', and their many variations with and without members and methods. By default, the DLR would return a simplified version of that chain of operations, which is not valid for our purposes. DynamicParser solves these scenarios thanks to both the invalidation mechanism described above, and to the way that the responsibility of binding dynamic operations is splited between the DynamicObject overrides, and the DynamicMetaObject ones.

Considerations

Parsing dynamic lambda expressions is not fast. Firstly, no surprises, the DLR itself is slower than regular C# code. And, secondly, the DynamicParser invalidation, last node, and surrogate mechanisms also carry their own load.

The time required to parse a dynamic lambda expression increases as the complexity of that expression increases. This is often not a problem because, firstly, these expressions tend to be simple, and secondly, because this time is typically much less than the ones required by other elements (such as network latencies in accessing databases). In any case, you should be aware of it because of its potential performance impact.

Be careful if you want to cache DynamicParser results. As explained above, it works by actually executing the dynamic lambda expressions. So, if any part of the expression invokes an external method, then the value obtained is the one at the moment when the expression is parsed/executed.

In layman terms, if you try to cache parameterized expressions, as for instance database ones, it won't work by default. The value of these parameters will always be the one captured at the time when the expression was parsed. Remember that DynamicNode instance are immutable ones.

Having said that, it is not difficult to implement a visitor-alike solution that rewrites the appropriate parts of a DynamicNode results' chain. This is actually part of the backlog for future versions.

Limitations

Coalesce Operations Do Not Work. Dynamic Lambda Expressions such as 'x => x.Alpha ?? x.Beta' resolves straight into the left operand, x.Alpha in the example.

The interesting bit here is that the dynamic operation invoked when the dynamic lambda expression is executed is just a get member operation, but it is not then followed by a binary or comparison one.

The overriden DLR-related methods in the library are not even invoked.

Ternary Conditional Operations Do Not Work. Dynamic Lambda Expressions such as 'x => x.Alpha ? x.Beta : x.Delta' resolves into the last operand: x.Delta in the example.

Remember that DynamicParser works by executing the Dynamic Lambda. In this case, the method in charge of intercepting unary operations is invoked to parse what essencially is a 'IsTrue(x.Alpha)' operation. DynamicParser returns a false value because the DLR is expecting a boolean value as the result (we could have chosen true as well). But the net result is that the DLR chooses the second branch, hence returning the last operand.

Note that the ternary conditional operation is essentially a compiler trick, not a fundamental operator (in the sense of implementation). DynamicParser has currently no way of understanding this is ternary conditional operation instead of an unary one, and so no way of invoking it twice (one for the false branch, and then another for the true one).

Testing

To validate that the DLR rules are not reused, you will notice the tests contain the same member and method names over and over again. This is not a mistake but, rather, is done on purpose so that we validate we are preventing the DLR to reuse previous rules and intermediate results.

I have also noticed we shall also take in consideration what test engine we are using (in my case, Xunit). I've found that, when instructing Xunit to run tests in parallel, each is run in a kind-of private DLR context that don't reuse previous DLR rules, and passing all the test cases with no invalidations invoked. . As our intention is precissely the opposite (having rules to invalidate), it has forced me to write a console application that finds and executes these test methods in sequence.

History

The family of Yotei libraries inherit from the previous work done for the Kerosene one, which still can be found in Code Project and other sources.

[0.5.6]: Current version of the package. Includes support for dynamic conversion operations.

[0.2.0]: Almost a straight port from the Kerosene version, but using the newest .NET version.