Earl.Crawler 0.0.0-alpha.0.65

This is a prerelease version of Earl.Crawler.

There is a newer prerelease version of this package available.
See the version list below for details.

dotnet add package Earl.Crawler --version 0.0.0-alpha.0.65

NuGet\Install-Package Earl.Crawler -Version 0.0.0-alpha.0.65

This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.

<PackageReference Include="Earl.Crawler" Version="0.0.0-alpha.0.65" />

For projects that support PackageReference, copy this XML node into the project file to reference the package.

paket add Earl.Crawler --version 0.0.0-alpha.0.65

The NuGet Team does not provide support for this client. Please contact its maintainers for support.

#r "nuget: Earl.Crawler, 0.0.0-alpha.0.65"

#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.

// Install Earl.Crawler as a Cake Addin
#addin nuget:?package=Earl.Crawler&version=0.0.0-alpha.0.65&prerelease

// Install Earl.Crawler as a Cake Tool
#tool nuget:?package=Earl.Crawler&version=0.0.0-alpha.0.65&prerelease

The NuGet Team does not provide support for this client. Please contact its maintainers for support.

Earl

Earl is a suite of APIs for developing url crawlers & web scrapers driven by a middleware pattern similar to, and strongly influenced by, ASP.NET Core.

Basic Usage

var services = new ServiceCollection()
    .AddEarlCrawler()
    .AddEarlJsonPersistence()
    .BuildServiceProvider();

var crawler = services.GetService<IEarlCrawler>();
var options = CrawlerOptionsBuilder.CreateDefault()
    .BatchSize( 50 )
    .MaxRequestCount( 500 )
    .On<CrawlUrlResultEvent>( 
        ( CrawlUrlResultEvent e, CancellationToken cancellation ) =>
        {
            Console.WriteLine( $"Crawled {e.Result.Url}" );
            return ValueTask.CompletedTask;
        }
    )
    .Timeout( TimeSpan.FromMinutes( 30 ) )
    .Use(
        ( CrawlUrlContext context, CrawlUrlDelegate next ) =>
        {
            Console.WriteLine( $"Executing delegate middleware while crawling {context.Url}" );
            return next( context );
        }
    )
    .PersistTo( persist => persist.ToJson( json => json.Destination(...) ) )
    .Build();

await crawler.CrawlAsync( new Uri(...), options );

Documentation

Documentation can be find within the READMEs of the sub-directories representing the conceptual components of Earl:

All public APIs should contain thorough XML (triple slash) comments.

Something missing, still have questions? Please open an Issue or submit a PR!

Product	Compatible and additional computed target framework versions.
.NET	net6.0 is compatible. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 was computed. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed.

Product

.NET

Compatible target framework(s)

Included target framework(s) (in package)

Learn more about Target Frameworks and .NET Standard.

net6.0
- Earl.Crawler.Abstractions (>= 0.0.0-alpha.0.65)
- Earl.Crawler.Events (>= 0.0.0-alpha.0.65)
- Earl.Crawler.Middleware (>= 0.0.0-alpha.0.65)
- Earl.Crawler.Middleware.Html (>= 0.0.0-alpha.0.65)
- Earl.Crawler.Middleware.Http (>= 0.0.0-alpha.0.65)
- Earl.Crawler.Middleware.UrlScraping (>= 0.0.0-alpha.0.65)

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.