Rhetos.ElasticSearch 6.1.0

Rhetos.ElasticSearch

Rhetos.ElasticSearch is a DSL extension for Rhetos development platform. It provides an implementation of filtering data via external OpenSearch or Elasticsearch engine.

Contents:

1. Installation
2. Configuration
3. Usage
   1. Creating ElasticSearch index
   2. Creating search function
   3. Command Line Interface
   4. RhetosElasticClient
4. How to contribute
   1. Building and testing the source code

See rhetos.org for more information on Rhetos.

Installation

Installing this package to a Rhetos application:

1. Add "Rhetos.ElasticSearch" NuGet package, available at the NuGet.org on-line gallery.
2. In Startup.ConfigureServices method, extend the Rhetos services configuration (at services.AddRhetosHost) with: .AddElasticSearch()
3. For updating Elasticsearch index in background, it is required to install and configure a Rhetos.Jobs implementations package, for example Rhetos.Jobs.Hangfire.

For backward compatibility, the packages name and method names are based on Elasticsearch, but when work with both OpenSearch and Elasticsearch search engines.

To run OpenSearch or Elasticsearch server for testing, you can use docker containers

Configuration

Configuration of the plugin is done in app settings, for example in rhetos-app.local.settings.json file.

For backward compatibility, the configuration section is named ElasticSearch, but it applies to both OpenSearch and Elasticsearch search engines.

{
  "Rhetos": {
    "ElasticSearch": {
      "SearchEngine": "Elasticsearch", //Elasticsearch/OpenSearch/WriteBothReadElasticsearch/WriteBothReadOpenSearch. Writing to both servers allows active transitional period when migrating from one to the other.
      "ServerUrls": ["https://ElasticSearchServer1", "https://ElasticSearchServer2"], //List of URLs of the ElasticSearch server instances.
      "ServerUrls_Elastic": ["https://ElasticSearchServer1", "https://ElasticSearchServer2"], //Use instead of ServerUrls, when SearchEngine is WriteBothReadElasticsearch or WriteBothReadOpenSearch.
      "ServerUrls_OpenSearch": ["https://OpenSearchServer1", "https://OpenSearchServer2"], //Use instead of ServerUrls, when SearchEngine is WriteBothReadElasticsearch or WriteBothReadOpenSearch.
      "UserName": "admin", //Optional parameter. If provided in combination with password, BasicAuthentication will be used.
      "Password": "admin", //Optional parameter. 
      "EnvironmentOverride": "ElasticSearchTest", //Optional parameter. Name of the application instance. Used for constructing index name. If not specified environment is constructed from server and database name retrieved from Rhetos connection string.
      "DefaultQuerySize": 20, //This is default value of the parameter.
      "MaxQuerySize": 1000, //This is default value of the parameter.
      "IndexSaveWaitSeconds": 300, //This is default value of the parameter.
      "IndexElasticBatchSize": 20000, //This is default value of the parameter.
      "ReindexMaxBatchSize": 300000, //This is default value of the parameter. Max batch when reindexing entire index using `esc.exe /r`. Max batch size for populating index queue is multiplied by 10.
      "NumberOfShards": 0, //This is default value of the parameter. Value 0 means that number of shards will be taken from number of nodes, otherwise number of shards will be as value provided.
      "NumberOfReplicas": 0, //This is default value of the parameter. For production purposes you will want to have at least 1 replica.
      "PatternAnalyzerDataRegex": "\d\w*([/\-,\.]\d\w*)+|\w+", //This is default value of the parameter. Regex used for pattern analyzer in elastic index.
      "PatternAnalyzerFilterRegex": "\d\w*([/\-,\.]\d\w*)+|\w+", //This is default value of the parameter. Regex used for preprocesing filter on property that has pattern analyzer defined.
    }
  }
}

ICU Analysis Plugin needs to be installed on the OpenSearch or Elasticsearch server.

You can use docker-compose as an easy way to run OpenSearch or Elasticsearch server with ICU Analysis plugin on development and test environment, see the files in Tools/TestService.

Usage

For the example purposes following entity will be used:

Module ElasticTest
{
  Entity Catalog
  {
    ShortString Name;
  }
  
  Entity TheEntity
  {
    ShortString Name;
    LongString Description;
    Reference Catalog;
    Date TheDate;
    LongString SubItemIds;
    Integer TheInt;
    Decimal TheDecimal;
    Money TheMoney;
    Bool TheBool;
    DateTime TheDateTime;
  }
}

Creating OpenSearch or Elasticsearch index

For backward compatibility, the keywords are based on Elasticsearch naming, but whey applies to both OpenSearch and ElasticSearch engines.

ElasticIndex concept is used to create data source for populating index. It also creates:

• Entity [ElasticIndexName]Queue - represents a queue for syncing data to index in OpenSearch or Elasticsearch.
• Action [ElasticIndexName]QueueAllItems - queues all items into queue in order to re-index all of the data.
• Action [ElasticIndexName]SyncIndex - creates index in OpenSearch or Elasticsearch (if not already created) and syncs queued data to the index.
• Action [ElasticIndexName]SyncItems - creates index in OpenSearch or Elasticsearch (if not already created) and syncs items which ids are passed in the ItemIds parameter of the action. Action also removes passed ItemIds from [ElasticIndexName]Queue on success.

ElasticIndex concept is derived from Browse concept so you can use Take concepts for adding wanted properties. For each property you can define how that property will be searched. For that purpose number of 'ElasticWhere' concepts are defined:

• ElasticWhere - this concept is used for ShortString, LongString and Bool properties.
• ElasticWhereAny - this concept is used for Guid and Reference properties.
• ElasticWhereAll - this concept is used for LongString properties that contains number of GUIDs separated by spaces.
• ElasticWhereRange - this concept is used for date and number properties.

Here is the example of creating OpenSearch or Elasticsearch index:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    QueueOrder "TheDateTime DESC"; // Optional sort for queue when indexing entire elastic index.

    Take Name { ElasticWhere; }
    Take Description; //Just add property to index. We will add custom search for this property later.
    Take Catalog { ElasticWhereAny; }
    Take CatalogName 'Catalog.Name' { ElasticWhere; } 
    Take TheDate { ElasticWhereRange; }
    Take TheDateTime { ElasticWhereRange; }
    Take SubItemIds { ElasticWhereAll;  }
    Take TheDecimal { ElasticWhereRange; }
    Take TheMoney { ElasticWhereRange; }
    Take TheInt { ElasticWhereRange; }
    Take TheBool { ElasticWhere; }
  }

ElasticIndex concept also has built in functionality of automatically inserting items into queue when source item is inserted or updated. If you want additional dependency entities to be included in this functionality simply add ChangesOnChangedItems accordingly:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Name { ElasticWhere; }
    Take Description; //Just add property to index. We will add custom search for this property later.
    Take Catalog { ElasticWhereAny; }
    Take CatalogName 'Catalog.Name' { ElasticWhere; } 
    Take TheDate { ElasticWhereRange; }
    Take TheDateTime { ElasticWhereRange; }
    Take SubItemIds { ElasticWhereAll;  }
    Take TheDecimal { ElasticWhereRange; }
    Take TheMoney { ElasticWhereRange; }
    Take TheInt { ElasticWhereRange; }
    Take TheBool { ElasticWhere; }

    ChangesOnChangedItems ElasticTest.Catalog 'FilterCriteria' 'changedItems =>
      {
        var ids = changedItems.Select(item => item.ID).Distinct();
        return new FilterCriteria("CatalogID", "in", ids);
      }';
  }

You can also use SyncOnChangedItems which replaces ChangesOnChangedItems and gives additional functionality of choosing Hangfire queue under which Elastic Index synchronization will be executed:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Name { ElasticWhere; }
    Take Description; //Just add property to index. We will add custom search for this property later.
    Take Catalog { ElasticWhereAny; }
    Take CatalogName 'Catalog.Name' { ElasticWhere; } 
    Take TheDate { ElasticWhereRange; }
    Take TheDateTime { ElasticWhereRange; }
    Take SubItemIds { ElasticWhereAll;  }
    Take TheDecimal { ElasticWhereRange; }
    Take TheMoney { ElasticWhereRange; }
    Take TheInt { ElasticWhereRange; }
    Take TheBool { ElasticWhere; }

    SyncOnChangedItems ElasticTest.Catalog 'FilterCriteria' 'changedItems =>
      {
        var ids = changedItems.Select(item => item.ID).Distinct();
        return new FilterCriteria("CatalogID", "in", ids);
      }' 'critical_elastic';
	  
	  // or

    SyncOnChangedItems ElasticTest.Catalog 'FilterCriteria' 'changedItems =>
      {
        var ids = changedItems.Select(item => item.ID).Distinct();
        return new FilterCriteria("CatalogID", "in", ids);
      }'; // When Hangfire queue is not specified default value is used ('default_elastic')
  }

Just bare in mind when you use SyncOnChangedItems you will have to add adittional queue names into Rhetos.Jobs options:

"Queues": ["default", "critical_elastic", "default_elastic", "low_priority_elastic"]

Also, you can add your code snippet into function which is called every time number of items are enqueued for indexing by declaring it via AfterEnqueueForIndex concept:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Name { ElasticWhere; }
    Take Description; //Just add property to index. We will add custom search for this property later.
    Take Catalog { ElasticWhereAny; }
    Take CatalogName 'Catalog.Name' { ElasticWhere; } 

    AfterEnqueueForIndex 'Custom code that will be executed after enqueue'
    '
        //here goes the code
        _backgroundJobs.EnqueueAction(new MyModule.MyAction
        {
            FilterObjectJson = filterObjectJson
        }, false, true, "default");
    ';
  }

For ShortString and LongString properties you can use number of analyzers for populating elastic index. You can use any of OpenSearch or Elasticsearch built-in analyzers. There are also two additional analyzers provided:

• whitespace_lowercase - uses built-in whitespace tokenizer and converts text to lowercase
• pattern_lowercase - uses built-in pattern tokenizer (with regex provided in configuration file - ElasticSearch:PatternAnalyzerDataRegex) and converts text to lowercase. This analyzer also implements IFilterParser interface which is called for filter string preprocessing with regex provided in configuration file (ElasticSearch:PatternAnalyzerFilterRegex)

You can assign analyzer for certain property like this:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Description { ElasticWhere; ElasticAnalyzer "standard"; } //you can use any of the Elastic built in or custom analyzers
  }

You can also register your own analyzer and then use it. Here is an example of custom analyzers:

  public class WhitespaceLowerCaseAnalyzer : ICustomElasticAnalyzer
  {
    public string Name => "whitespace_lowercase";
    public Func<CustomAnalyzerDescriptor, ICustomAnalyzer> Analyzer => descriptor => descriptor.Tokenizer("whitespace").Filters("lowercase");
    public Func<TokenizersDescriptor, TokenizersDescriptor> Tokenizer => null;
  }

or like this when your analyzer also needs to preprocess filter string

  public class CustomPatternAnalyzer : ICustomElasticAnalyzer, IFilterParser
  {
    public string Name => "custom_pattern_analyzer";
    public Func<CustomAnalyzerDescriptor, ICustomAnalyzer> Analyzer => descriptor => descriptor.Tokenizer("custom_pattern_tokenizer").Filters("lowercase");
    public Func<TokenizersDescriptor, TokenizersDescriptor> Tokenizer => descriptor =>  
      descriptor.Pattern("custom_pattern_tokenizer", p => p.Pattern(@"\d\w*([/\-,\.]\d\w*)+|UP/II?[/\-\d]*|\w+").Group(0));

  public string ParseFilter(string filter)
    {
      if (string.IsNullOrWhiteSpace(filter))
        return filter;
      var tokensRegex = new Regex(@"\d\w*([/\-,\.]\d\w*)+|UP/II?[/\-\d]*|\w+", RegexOptions.IgnoreCase);
      return string.Join(" ", tokensRegex.Matches(filter).Cast<Match>().Select(x => x.Value).ToList());
    }
  }

Once you have created your analyzer you have to register it in Autofac:

  [Export(typeof(Module))]
  public class AutofacModuleConfiguration : Module
  {
    protected override void Load(ContainerBuilder builder)
    {
      builder.RegisterType<WhitespaceLowerCaseAnalyzer>().As<ICustomElasticAnalyzer>().SingleInstance();
      builder.RegisterType<CustomPatternAnalyzer>().As<ICustomElasticAnalyzer>().SingleInstance();
      //When analyzer implements IFilterParser interface it also needs to be registered directly  
      builder.RegisterType<CustomPatternAnalyzer>().SingleInstance();
      base.Load(builder);
    }
  }

And now you can us it in your DSL:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Description { ElasticWhere; ElasticAnalyzer "custom_pattern_analyzer"; } //you can use any of the Elastic built in analyzers
  }

Creating search function

Once ElasticIndex is created it can be used on any Browse or SqlQueryable for the same source Entity. To create search function ElasticSearch concept is used:

  Browse TheEntityBrowse ElasticTest.TheEntity
  {
    Take Name;
    Take Description;
    Take Catalog;
    Take CatalogName 'Catalog.Name';
    Take TheDate;
    
    ElasticSearch ElasticTest.TheEntityElastic;
  }

ElasticSearch concept creates search function named [BrowseName]Search. For each property in ElasticIndex that has 'ElasticWhere' defined, specific function parameter(s) are created:

• ElasticWhere - parameter with same name and type is created
• ElasticWhereAny - parameter with name [PropertyName]List and type List<Guid?> is created.
• ElasticWhereAll - parameter with name [PropertyName]IDList and type List<Guid?> is created.
• ElasticWhereRange - two parameters of the same type and names [PropertyName]From and [PropertyName]To are created.

In each search function there is also Ids property of type List<Guid?> in order to retrieve only exact items.

Along with search parameters, sorting and paging parameters are created:

• Sort - string, supports property name and sort direction, for example "Name" or "TheDate DESC"
• Skip - integer
• Take - integer

ElasticSearch concept also creates return type for search function named [BrowseName]SearchResult. Return type is a DataStructure with following properties:

• Data - List<TheEntityBrowse> for the example above.
• Total - int that represents number of items which satisfies search parameters.

Now we will add custom search for Description property for example above. For that purpose AdditionalElasticQuery concept is used.

  ShortString TheEntityBrowseSearch TheDescription; //add parameter to search function
  
  Browse TheEntityBrowse ElasticTest.TheEntity
  {
    ElasticSearch ElasticTest.TheEntityElastic
    {
      AdditionalElasticQuery 'Custom additional filter' '
        if (!string.IsNullOrWhiteSpace(parameter.TheDescription))
          query.Where(q => q.Where(f => f.Description, parameter.TheDescription));
      ';      
    }
  }

In your custom code you can use following provided variables:

• parameter - instance of search function parameters
• repository - instance of DomRepository
• _executionContext - instance of ExecutionContext
• userInfo - instance of IUserInfo
• query - instance of elastic query builder

One of the main purposes of custom search functionality is implementation of row permissions, since using RowPermissions concepts with ElasticSearch concept is impossible. Idea is to put data used for row permissions directly into index and then filter by that data. Here is an example:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Name { ElasticWhere; }
    Take CatalogID;
    Take CatalogName { ElasticWhere; }
    Take TheDate { ElasticWhereRange; }
    Take AllowedUsers 'Base.Extension_RowPermissionsQueryable.AllowedUsers' { ElasticWhereAll; }
  }

  Browse TheEntityBrowse ElasticTest.TheEntity
  {
    Take Name;
    Take Description;
    Take CatalogName 'Catalog.Name';
    Take TheDate;

    ElasticSearch ElasticTest.TheEntityElastic
    {
      AdditionalElasticQuery 'Row Permissions' '
        var userId = repository.Common.Principal.Query(x => x.Name == _executionContext.UserInfo.UserName).Single().ID;
        var allowedCatalogIds = repository.ElasticTest.AllowedCatalogs.Query(x => x.UserID == userId).Select(x => x.CatalogID).ToList();
        query.Where(q => q.WhereAll(f => f.AllowedUsers, new[]{userId}) || q.WhereAny(f => f.CatalogID), allowedCatalogIds);
      ';      
    }
  }

ElasticSearch concept can also be used on ElasticIndex directly. That way additional SQL query to retrieve filtered data is omitted since it is the same data that is stored in OpenSearch or Elasticsearch. Drawback of this approach is that when new property is added to the ElasticIndex entire index must be reindexed in order to return that new property. Here is an example:

  ElasticIndex TheEntityElastic ElasticTest.TheEntity
  {
    Take Name { ElasticWhere; }
    Take Description; //We do not want to search this property, just to return it in the result.
    Take Catalog { ElasticWhereAny; }
    Take CatalogName 'Catalog.Name' { ElasticWhere; } 
    Take TheDate { ElasticWhereRange; }

    ElasticSearch ElasticTest.TheEntityElastic;
  }

Command Line Interface

For manual indexing/reindexing CLI tool is provided. esc.exe is placed in Plugin folder of the Rhetos server. CLI switches are:

• /Index:[NameOfElasticIndex] or /I:[NameOfElasticIndex] - name of the index that should be used. For example /I:MyModule.MyElasticIndex
• /Del or /D - deletes index in OpenSearch or Elasticsearch.
• /Queue or /Q - queues all the items from the source to queue for syncing to OpenSearch or Elasticsearch.
• /Sync or /S - syncs all queued items to OpenSearch or Elasticsearch.
• /Reindex or /R - queues and syncs all items to OpenSearch or Elasticsearch. This command is convenient for consecutive call of above three commands.
• /V - prints additional data in console during long operations.

RhetosElasticClient

In RhetosElasticClient class is key implementation for communication with OpenSearch or Elasticsearch engine. Class is registered in Rhetos and you can use it in your code for example in your DOM code:

var elasticClient = container.Resolve<Rhetos.ElasticSearch.RhetosElasticClient>();

Or you can use it in your DSL like this:

Module Common
{
  Action ElasticIndexItems
  '(parameters, repository, userInfo) =>
  {
    _elasticClient.IndexItems(parameters.IndexName, parameters.ItemIds);
  }'
  {
    ShortString IndexName;
    ListOf Guid ItemIds;
    RepositoryUses '_elasticClient' 'Rhetos.ElasticSearch.RhetosElasticClient';
  }
}

Methods provided in RhetosElasticClient are:

• SearchResult<T> Search<T>(ElasticQuery<T> query) - method for searching the OpenSearch or Elasticsearch index.
• IndexItems(string elasticIndexName, IEnumerable<Guid> ids) - method for indexing items with specified ids. There are also following extension methods provided: IndexItems(string elasticIndexName, IEnumerable<Guid?> ids), IndexItems<TIndex>(IEnumerable<Guid> ids) and IndexItems<TIndex>(IEnumerable<Guid?> ids)
• IndexQueuedItems(string elasticIndexName, int? batchSize = null) - method for indexing one batch of items that are queued. If batchSize is omitted the value of configuration parameter ElasticSearch:IndexRhetosBatchSize is used. There is also extension method provided IndexQueuedItems<TIndex>(int? batchSize = null)
• DeleteIndex(string elasticIndexName) - method deletes entire index from OpenSearch or Elasticsearch. Use this method only for testing purposes or when structure of index is dramatically changed. There is also extension method provided DeleteIndex<TIndex>()

How to contribute

Contributions are very welcome. The easiest way is to fork this repo, and then make a pull request from your fork. The first time you make a pull request, you may be asked to sign a Contributor Agreement. For more info see How to Contribute on Rhetos wiki.

Building and testing the source code

• Note: This package is already available at the NuGet.org online gallery. You don't need to build it from source in order to use it in your application.
• Build:
  • To build the package from source, run Clean.bat and Build.bat.
  • The build output is a NuGet package in the "Install" subfolder.
• Testing:
  • For the test script to work, you need to create an empty database and a settings file test\TestApp\rhetos-app.local.settings.json with the database connection string (configuration key "ConnectionStrings:RhetosConnectionString").
  • Additionally you need OpenSearch and Elasticsearch servers. If you have Docker Desktop available, the easiest solution is to execute run.bat script in Tools\TestService, that will download and run the official containers with ICU plugin installed. Set the options ServerUrls_Elastic and ServerUrls_OpenSearch in rhetos-app.local.settings.json (see the run.bat script for URLs, or Configuration section above).
  • To run unit tests execute Test.bat.