Downloads of v 2.3.5544.21265


Average downloads per day


Last published

Share on Social Networks

Crawler-Lib Engine


The Crawler-Lib Engine is a general purpose workflow enabled task processor. It has evolved from a web crawler over data mining and information retrieval. It is throughput optimized and can perform thousands of tasks per second on standard hardware. Due to its workflow capabilities it allows to structure and parallelize even complex kind of work. Please visit the project page for the complete view of the Crawler-Lib Engine.

A license for the Anonymous Edition is included in the package. A license for the more powerful free Community Edition can be generated on the project page. A unrestricted license is available too.

To install Crawler-Lib Engine, run the following command in the Package Manager Console

Install-Package CrawlerLib.Engine -Version 2.3.5544.21265

Release Notes

Current Version
      HttpResonse.CharacterSet has now a setter, where the encoding can be changed after the request is analyzed.

      The TaskResultBase has a new method Process() where result processing can be implemented by overriding.
      It is possible to let the engine call this method ether directly or with a TPL Task. This is controlled with the property TaskResultProcessing
      which is by default TaskResultProcessingEnum.EnqueueFinishedTasks.

      Version 2.3.5538.27074
      The Retry workflow element supports SetRetryWork() and ResetRetryWork() within any handler of the child objects.

      Version 2.3.5535.18986
      Fixed: Wrong parenting in Retry workflow element

      Version 2.3.5529.650
      Introduced some header properties in HttpRequest

      Version 2.3.5515.2909
      Fixed: Crawler blocks after 600 tasks in the community edition
      Breaking: AddLimiter() takes name form LimiterConfig now.
      Minor fixes: Calculate workflow element, Quota throughput limiter

      Version 2.3.5513.24819
      * New Workflow Element:  Calculate - allows th assemble a result from multiple parallel parts (like a Group but with a result)
      * Fixed: Workflow parents which don't start children blocked task

      Version 2.3.5507.444
      * Refactoring: Removed DNS, TickTimestamp, TickTimeSpan

      Version 2.3.5503.33731
      * Fixed and extended 'Work' workflow element

      Version 2.3.5503.4086
      * Breaking: Replaces Licensing system, doesn't work with old licenses. Generate new ones on the Crawler-Lib homepage.

      Version 2.2.5490.21676
      * Fixed ClickOnce installer problems

      Version 2.2.5367.17954
      * Fix parenting error in Retry element
      * Unrestricted License available

      Version 2.1.5291.40524
      * Added additional constructors to several workflow elements. So you can construct and use them without specifying a complete configuration object for the element.
      * Added AwaitProcessingEnum awaitProcessing to several workflow element constructors, so you can specify that the continuation will be called on failure and check the Success property to decide what to do.

      Version 2.1.5288.13218:
      * The workflow elements are awaitable since this release.
      * New workflow elements for limits and operation cost calculation have been added.
      * A vast amount of small extensions and refactoring

      Version 2.00 -2.01 -2.02:
      * First public releases



Thomas Maierhofer


Copyright by Thomas Maierhofer



Version History

Version Downloads Last updated
Crawler-Lib Engine 2.3.5544.21265 (this version) 582 Saturday, March 7, 2015
Crawler-Lib Engine 2.3.5538.27074 121 Sunday, March 1, 2015
Crawler-Lib Engine 2.3.5535.18986 130 Thursday, February 26, 2015
Crawler-Lib Engine 2.3.5529.650 129 Friday, February 20, 2015
Crawler-Lib Engine 2.3.5515.2909 159 Friday, February 6, 2015
Crawler-Lib Engine 2.3.5514.24037 125 Thursday, February 5, 2015
Crawler-Lib Engine 2.3.5513.24819 133 Wednesday, February 4, 2015
Crawler-Lib Engine 2.3.5507.444 133 Thursday, January 29, 2015
Crawler-Lib Engine 2.3.5503.33731 137 Sunday, January 25, 2015
Crawler-Lib Engine 2.3.5503.4086 135 Sunday, January 25, 2015
Crawler-Lib Engine 2.2.5490.21676 164 Monday, January 12, 2015
Crawler-Lib Engine 2.2.5367.17954 198 Thursday, September 11, 2014
Crawler-Lib Engine 2.1.5296.19510 194 Wednesday, July 2, 2014
Crawler-Lib Engine 2.1.5291.40524 151 Friday, June 27, 2014
Crawler-Lib... 2.1.5288.13218 175 Tuesday, June 24, 2014
Crawler-Lib... 2.0.2 548 Friday, September 6, 2013
CrawlerLib.NET... 2.0.1 291 Tuesday, June 25, 2013
CrawlerLib.NET... 2.0.0 227 Saturday, June 22, 2013