andolaus 

PageProbe is a modern, extensible .NET 8 web crawling library for extracting links, multimedia, metadata, images, and text from web pages. It supports robust crawling with depth control, robots.txt compliance,... More information