Rhetos.ElasticSearch 6.0.0

Rhetos.ElasticSearch

Rhetos.ElasticSearch is a DSL extension for Rhetos development platform. It provides an implementation of filtering data via external ElasticSearch engine.

Contents:

1. Installation
2. Configuration
3. Usage
   1. Creating ElasticSearch index
   2. Creating search function
   3. Command Line Interface
   4. RhetosElasticClient
4. How to contribute
   1. Building and testing the source code

See rhetos.org for more information on Rhetos.

Installation

Installing this package to a Rhetos application:

1. Add "Rhetos.ElasticSearch" NuGet package, available at the NuGet.org on-line gallery.
2. In Startup.ConfigureServices method, extend the Rhetos services configuration (at services.AddRhetosHost) with: .AddElasticSearch()
3. For updating ElasticSearch index in background, it is required to install and configure a Rhetos.Jobs implementations package, for example Rhetos.Jobs.Hangfire.

Configuration

Configuration of the plugin is done in app settings, for example in rhetos-app.local.settings.json file.

{
  "Rhetos": {
    "ElasticSearch": {
      "ServerUrls": ["https://ElasticSearchServer1", "https://ElasticSearchServer2"], //Mandatory parameter. List of URLs of the ElasticSearch server instances.
      "UserName": "admin", //Optional parameter. If provided in combination with password, BasicAuthentication will be used.
      "Password": "admin", //Optional parameter. 
      "EnvironmentOverride": "ElasticSearchTest", //Optional parameter. Name of the application instance. Used for constructing index name. If not specified environment is constructed from server and database name retrieved from Rhetos connection string.
      "DefaultQuerySize": 20, //This is default value of the parameter.
      "MaxQuerySize": 1000, //This is default value of the parameter.
      "IndexSaveWaitSeconds": 300, //This is default value of the parameter.
      "IndexElasticBatchSize": 20000, //This is default value of the parameter.
      "ReindexMaxBatchSize": 300000, //This is default value of the parameter. Max batch when reindexing entire index using `esc.exe /r`. Max batch size for populating index queue is multiplied by 10.
      "NumberOfShards": 0, //This is default value of the parameter. Value 0 means that number of shards will be taken from number of nodes, otherwise number of shards will be as value provided.
      "NumberOfReplicas": 0, //This is default value of the parameter. For production purposes you will want to have at least 1 replica.
      "PatternAnalyzerDataRegex": "\d\w*([/\-,\.]\d\w*)+|\w+", //This is default value of the parameter. Regex used for pattern analyzer in elastic index.
      "PatternAnalyzerFilterRegex": "\d\w*([/\-,\.]\d\w*)+|\w+", //This is default value of the parameter. Regex used for preprocesing filter on property that has pattern analyzer defined.
    }
  }
}

ICU Analysis Plugin needs to be installed on the Elasticsearch server.

Usage

For the example purposes following entity will be used:

Module ElasticTest
{
  Entity Catalog
  {
    ShortString Name;
  }
  
  Entity TheEntity
  {
    ShortString Name;
    LongString Description;
    Reference Catalog;
    Date TheDate;
    LongString SubItemIds;
    Integer TheInt;
    Decimal TheDecimal;
    Money TheMoney;
    Bool TheBool;
    DateTime TheDateTime;
  }
}

Creating ElasticSearch index

ElasticIndex concept is used to create data source for populating index. It also creates:

• Entity [ElasticIndexName]Queue - represents a queue for syncing data to index in ElasticSearch.
• Action [ElasticIndexName]QueueAllItems - queues all items into queue in order to re-index all of the data.
• Action [ElasticIndexName]SyncIndex - creates index in ElasticSearch (if not already created) and syncs queued data to the index.
• Action [ElasticIndexName]SyncItems - creates index in ElasticSearch (if not already created) and syncs items which ids are passed in the ItemIds parameter of the action. Action also removes passed ItemIds from [ElasticIndexName]Queue on success.

ElasticIndex concept is derived from Browse concept so you can use Take concepts for adding wanted properties. For each property you can define how that property will be searched. For that purpose number of 'ElasticWhere' concepts are defined:

• ElasticWhere - this concept is used for ShortString, LongString and Bool properties.
• ElasticWhereAny - this concept is used for Guid and Reference properties.
• ElasticWhereAll - this concept is used for LongString properties that contains number of GUIDs separated by spaces.
• ElasticWhereRange - this concept is used for date and number properties.

Here is the example of creating ElasticSearch index:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
	QueueOrder "TheDateTime DESC"; // Optional sort for queue when indexing entire elastic index.
	
    Take Name { ElasticWhere; }
    Take Description; //Just add property to index. We will add custom search for this property later.
    Take Catalog { ElasticWhereAny; }
    Take CatalogName 'Catalog.Name' { ElasticWhere; } 
    Take TheDate { ElasticWhereRange; }
    Take TheDateTime { ElasticWhereRange; }
    Take SubItemIds { ElasticWhereAll;  }
    Take TheDecimal { ElasticWhereRange; }
    Take TheMoney { ElasticWhereRange; }
    Take TheInt { ElasticWhereRange; }
    Take TheBool { ElasticWhere; }
  }

ElasticIndex concept also has built in functionality of automatically inserting items into queue when source item is inserted or updated. If you want additional dependency entities to be included in this functionality simply add ChangesOnChangedItems accordingly:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Name { ElasticWhere; }
    Take Description; //Just add property to index. We will add custom search for this property later.
    Take Catalog { ElasticWhereAny; }
    Take CatalogName 'Catalog.Name' { ElasticWhere; } 
    Take TheDate { ElasticWhereRange; }
    Take TheDateTime { ElasticWhereRange; }
    Take SubItemIds { ElasticWhereAll;  }
    Take TheDecimal { ElasticWhereRange; }
    Take TheMoney { ElasticWhereRange; }
    Take TheInt { ElasticWhereRange; }
    Take TheBool { ElasticWhere; }

    ChangesOnChangedItems ElasticTest.Catalog 'FilterCriteria' 'changedItems =>
      {
        var ids = changedItems.Select(item => item.ID).Distinct();
        return new FilterCriteria("CatalogID", "in", ids);
      }';
  }

You can also use SyncOnChangedItems which replaces ChangesOnChangedItems and gives additional functionality of choosing Hangfire queue under which Elastic Index synchronization will be executed:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Name { ElasticWhere; }
    Take Description; //Just add property to index. We will add custom search for this property later.
    Take Catalog { ElasticWhereAny; }
    Take CatalogName 'Catalog.Name' { ElasticWhere; } 
    Take TheDate { ElasticWhereRange; }
    Take TheDateTime { ElasticWhereRange; }
    Take SubItemIds { ElasticWhereAll;  }
    Take TheDecimal { ElasticWhereRange; }
    Take TheMoney { ElasticWhereRange; }
    Take TheInt { ElasticWhereRange; }
    Take TheBool { ElasticWhere; }

    SyncOnChangedItems ElasticTest.Catalog 'FilterCriteria' 'changedItems =>
      {
        var ids = changedItems.Select(item => item.ID).Distinct();
        return new FilterCriteria("CatalogID", "in", ids);
      }' 'critical_elastic';
	  
	  // or

    SyncOnChangedItems ElasticTest.Catalog 'FilterCriteria' 'changedItems =>
      {
        var ids = changedItems.Select(item => item.ID).Distinct();
        return new FilterCriteria("CatalogID", "in", ids);
      }'; // When Hangfire queue is not specified default value is used ('default_elastic')
  }

Just bare in mind when you use SyncOnChangedItems you will have to add adittional queue names into Rhetos.Jobs options:

"Queues": ["default", "critical_elastic", "default_elastic", "low_priority_elastic"]

Also, you can add your code snippet into function which is called every time number of items are enqueued for indexing by declaring it via AfterEnqueueForIndex concept:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Name { ElasticWhere; }
    Take Description; //Just add property to index. We will add custom search for this property later.
    Take Catalog { ElasticWhereAny; }
    Take CatalogName 'Catalog.Name' { ElasticWhere; } 

    AfterEnqueueForIndex 'Custom code that will be executed after enqueue'
    '
        //here goes the code
        _backgroundJobs.EnqueueAction(new MyModule.MyAction
        {
            FilterObjectJson = filterObjectJson
        }, false, true, "default");
    ';
  }

For ShortString and LongString properties you can use number of analyzers for populating elastic index. You can use any of ElasticSearch built-in analyzers. There are also two additional analyzers provided:

• whitespace_lowercase - uses built-in whitespace tokenizer and converts text to lowercase
• pattern_lowercase - uses built-in pattern tokenizer (with regex provided in configuration file - ElasticSearch:PatternAnalyzerDataRegex) and converts text to lowercase. This analyzer also implements IFilterParser interface which is called for filter string preprocessing with regex provided in configuration file (ElasticSearch:PatternAnalyzerFilterRegex)

You can assign analyzer for certain property like this:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Description { ElasticWhere; ElasticAnalyzer "standard"; } //you can use any of the Elastic built in or custom analyzers
  }

You can also register your own analyzer and then use it. Here is an example of custom analyzers:

  public class WhitespaceLowerCaseAnalyzer : ICustomElasticAnalyzer
  {
    public string Name => "whitespace_lowercase";
    public Func<CustomAnalyzerDescriptor, ICustomAnalyzer> Analyzer => descriptor => descriptor.Tokenizer("whitespace").Filters("lowercase");
    public Func<TokenizersDescriptor, TokenizersDescriptor> Tokenizer => null;
  }

or like this when your analyzer also needs to preprocess filter string

  public class CustomPatternAnalyzer : ICustomElasticAnalyzer, IFilterParser
  {
    public string Name => "custom_pattern_analyzer";
    public Func<CustomAnalyzerDescriptor, ICustomAnalyzer> Analyzer => descriptor => descriptor.Tokenizer("custom_pattern_tokenizer").Filters("lowercase");
    public Func<TokenizersDescriptor, TokenizersDescriptor> Tokenizer => descriptor =>  
      descriptor.Pattern("custom_pattern_tokenizer", p => p.Pattern(@"\d\w*([/\-,\.]\d\w*)+|UP/II?[/\-\d]*|\w+").Group(0));

  public string ParseFilter(string filter)
    {
      if (string.IsNullOrWhiteSpace(filter))
        return filter;
      var tokensRegex = new Regex(@"\d\w*([/\-,\.]\d\w*)+|UP/II?[/\-\d]*|\w+", RegexOptions.IgnoreCase);
      return string.Join(" ", tokensRegex.Matches(filter).Cast<Match>().Select(x => x.Value).ToList());
    }
  }

Once you have created your analyzer you have to register it in Autofac:

  [Export(typeof(Module))]
  public class AutofacModuleConfiguration : Module
  {
    protected override void Load(ContainerBuilder builder)
    {
      builder.RegisterType<WhitespaceLowerCaseAnalyzer>().As<ICustomElasticAnalyzer>().SingleInstance();
      builder.RegisterType<CustomPatternAnalyzer>().As<ICustomElasticAnalyzer>().SingleInstance();
      //When analyzer implements IFilterParser interface it also needs to be registered directly  
      builder.RegisterType<CustomPatternAnalyzer>().SingleInstance();
      base.Load(builder);
    }
  }

And now you can us it in your DSL:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Description { ElasticWhere; ElasticAnalyzer "custom_pattern_analyzer"; } //you can use any of the Elastic built in analyzers
  }

Creating search function

Once ElasticIndex is created it can be used on any Browse or SqlQueryable for the same source Entity. To create search function ElasticSearch concept is used:

  Browse TheEntityBrowse ElasticTest.TheEntity
  {
    Take Name;
    Take Description;
    Take Catalog;
    Take CatalogName 'Catalog.Name';
    Take TheDate;
    
    ElasticSearch ElasticTest.TheEntityElastic;
  }

ElasticSearch concept creates search function named [BrowseName]Search. For each property in ElasticIndex that has 'ElasticWhere' defined, specific function parameter(s) are created:

• ElasticWhere - parameter with same name and type is created
• ElasticWhereAny - parameter with name [PropertyName]List and type List<Guid?> is created.
• ElasticWhereAll - parameter with name [PropertyName]IDList and type List<Guid?> is created.
• ElasticWhereRange - two parameters of the same type and names [PropertyName]From and [PropertyName]To are created.

In each search function there is also Ids property of type List<Guid?> in order to retrieve only exact items.

Along with search parameters, sorting and paging parameters are created:

• Sort - string, supports property name and sort direction, for example "Name" or "TheDate DESC"
• Skip - integer
• Take - integer

ElasticSearch concept also creates return type for search function named [BrowseName]SearchResult. Return type is a DataStructure with following properties:

• Data - List<TheEntityBrowse> for the example above.
• Total - int that represents number of items which satisfies search parameters.

Now we will add custom search for Description property for example above. For that purpose AdditionalElasticQuery concept is used.

  ShortString TheEntityBrowseSearch TheDescription; //add parameter to search function
  
  Browse TheEntityBrowse ElasticTest.TheEntity
  {
    ElasticSearch ElasticTest.TheEntityElastic
    {
      AdditionalElasticQuery 'Custom additional filter' '
        if (!string.IsNullOrWhiteSpace(parameter.TheDescription))
          query.Where(q => q.Where(f => f.Description, parameter.TheDescription));
      ';      
    }
  }

In your custom code you can use following provided variables:

• parameter - instance of search function parameters
• repository - instance of DomRepository
• _executionContext - instance of ExecutionContext
• userInfo - instance of IUserInfo
• query - instance of elastic query builder

One of the main purposes of custom search functionality is implementation of row permissions, since using RowPermissions concepts with ElasticSearch concept is impossible. Idea is to put data used for row permissions directly into index and then filter by that data. Here is an example:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Name { ElasticWhere; }
    Take CatalogID;
    Take CatalogName { ElasticWhere; }
    Take TheDate { ElasticWhereRange; }
    Take AllowedUsers 'Base.Extension_RowPermissionsQueryable.AllowedUsers' { ElasticWhereAll; }
  }

  Browse TheEntityBrowse ElasticTest.TheEntity
  {
    Take Name;
    Take Description;
    Take CatalogName 'Catalog.Name';
    Take TheDate;

    ElasticSearch ElasticTest.TheEntityElastic
    {
      AdditionalElasticQuery 'Row Permissions' '
        var userId = repository.Common.Principal.Query(x => x.Name == _executionContext.UserInfo.UserName).Single().ID;
        var allowedCatalogIds = repository.ElasticTest.AllowedCatalogs.Query(x => x.UserID == userId).Select(x => x.CatalogID).ToList();
        query.Where(q => q.WhereAll(f => f.AllowedUsers, new[]{userId}) || q.WhereAny(f => f.CatalogID), allowedCatalogIds);
      ';      
    }
  }

ElasticSearch concept can also be used on ElasticIndex directly. That way additional SQL query to retrieve filtered data is omitted since it is the same data that is stored in ElasticSearch. Drawback of this approach is that when new property is added to the ElasticIndex entire index must be reindexed in order to return that new property. Here is an example:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Name { ElasticWhere; }
    Take Description; //We do not want to search this property, just to return it in the result.
    Take Catalog { ElasticWhereAny; }
    Take CatalogName 'Catalog.Name' { ElasticWhere; } 
    Take TheDate { ElasticWhereRange; }

    ElasticSearch ElasticTest.TheEntityElastic;
  }

Command Line Interface

For manual indexing/reindexing CLI tool is provided. esc.exe is placed in Plugin folder of the Rhetos server. CLI switches are:

• /Index:[NameOfElasticIndex] or /I:[NameOfElasticIndex] - name of the index that should be used. For example /I:MyModule.MyElasticIndex
• /Del or /D - deletes index in ElasticSearch.
• /Queue or /Q - queues all the items from the source to queue for syncing to ElasticSearch.
• /Sync or /S - syncs all queued items to ElasticSearch.
• /Reindex or /R - queues and syncs all items to ElasticSearch. This command is convenient for consecutive call of above three commands.
• /V - prints additional data in console during long operations.

RhetosElasticClient

In RhetosElasticClient class is key implementation for communication with ElasticSearch engine. Class is registered in Rhetos and you can use it in your code for example in your DOM code:

var elasticClient = container.Resolve<Rhetos.ElasticSearch.RhetosElasticClient>();

Or you can use it in your DSL like this:

Module Common
{
  Action ElasticIndexItems
  '(parameters, repository, userInfo) =>
  {
    _elasticClient.IndexItems(parameters.IndexName, parameters.ItemIds);
  }'
  {
    ShortString IndexName;
    ListOf Guid ItemIds;
    RepositoryUses '_elasticClient' 'Rhetos.ElasticSearch.RhetosElasticClient';
  }
}

Methods provided in RhetosElasticClient are:

• SearchResult<T> Search<T>(ElasticQuery<T> query) - method for searching the ElasticSearch index.
• IndexItems(string elasticIndexName, IEnumerable<Guid> ids) - method for indexing items with specified ids. There are also following extension methods provided: IndexItems(string elasticIndexName, IEnumerable<Guid?> ids), IndexItems<TIndex>(IEnumerable<Guid> ids) and IndexItems<TIndex>(IEnumerable<Guid?> ids)
• IndexQueuedItems(string elasticIndexName, int? batchSize = null) - method for indexing one batch of items that are queued. If batchSize is omitted the value of configuration parameter ElasticSearch:IndexRhetosBatchSize is used. There is also extension method provided IndexQueuedItems<TIndex>(int? batchSize = null)
• DeleteIndex(string elasticIndexName) - method deletes entire index from ElasticSearch. Use this method only for testing purposes or when structure of index is dramatically changed. There is also extension method provided DeleteIndex<TIndex>()

How to contribute

Contributions are very welcome. The easiest way is to fork this repo, and then make a pull request from your fork. The first time you make a pull request, you may be asked to sign a Contributor Agreement. For more info see How to Contribute on Rhetos wiki.

Building and testing the source code

• Note: This package is already available at the NuGet.org online gallery. You don't need to build it from source in order to use it in your application.
• To build the package from source, run Clean.bat, Build.bat and Test.bat.
• For the test script to work, you need to create an empty database and a settings file test\TestApp\rhetos-app.local.settings.json with the database connection string (configuration key "ConnectionStrings:RhetosConnectionString") and Rhetos.ElasticSearch configuration (see Configuration section above).
• The build output is a NuGet package in the "Install" subfolder.