CrawlOptions

import type { CrawlOptions } from "@pitlane/crawler";

What crawl fetches and how: the starting paths, whether to follow links and queue assets, how many requests run at once, and hooks for nofollow pages and redirects. Every option has a default, so crawl(router) walks everything reachable from /.

See

Crawler guide

Properties

assets?

optional assets?: boolean;

Queue the <link href>, <script src>, and <img src> a page references. Turn it off when something else already emitted those files, as a bundler does.

Default

true

concurrency?

optional concurrency?: number;

How many paths to fetch at once.

Default

1

ignorePageNofollow?

optional ignorePageNofollow?: (pathname: string) => boolean;

Crawl a page's links even though the page asked robots not to follow them. Receives the page's path.

Parameters

pathname

string

Returns

boolean

onRedirect?

optional onRedirect?: (pathname: string, location: string | null) => void;

Called for a path that answered with a redirect rather than a document. Nothing is yielded for it: there is no page to write, and the app still answers the path at runtime.

Receives the requested path and the Location it pointed at, which is null for a redirect that named none.

Parameters

pathname

string

location

string | null

Returns

void

paths?

optional paths?: string[];

Paths to start from.

Default

["/"]

spider?

optional spider?: boolean;

Follow <a href> and <link rel="alternate"> to discover more paths. Turn it off to fetch exactly the paths given.

Default

true