Pitlane

@pitlane/crawler ​

crawl() — spider a Remix 3 fetch router in memory. Requests go straight into router.fetch, so an app can be walked and written to disk without a socket, a server, or a browser. staticPaths() answers the question that comes first: which paths a route map can serve with no params.

Interfaces ​

CrawlOptions ​

Properties ​

assets? ​
ts
optional assets?: boolean;

Queue the <link href>, <script src>, and <img src> a page references. Turn it off when something else already emitted those files, as a bundler does.

Default ​
ts
true
concurrency? ​
ts
optional concurrency?: number;

How many paths to fetch at once.

Default ​
ts
1
ignorePageNofollow? ​
ts
optional ignorePageNofollow?: (pathname) => boolean;

Crawl a page's links even though the page asked robots not to follow them. Receives the page's path.

Parameters ​
pathname ​

string

Returns ​

boolean

onRedirect? ​
ts
optional onRedirect?: (pathname, location) => void;

Called for a path that answered with a redirect rather than a document. Nothing is yielded for it: there is no page to write, and the app still answers the path at runtime.

Receives the requested path and the Location it pointed at, which is null for a redirect that named none.

Parameters ​
pathname ​

string

location ​

string | null

Returns ​

void

paths? ​
ts
optional paths?: string[];

Paths to start from.

Default ​
ts
["/"]
spider? ​
ts
optional spider?: boolean;

Follow <a href> and <link rel="alternate"> to discover more paths. Turn it off to fetch exactly the paths given.

Default ​
ts
true

CrawlResult ​

One fetched path and the response the router produced for it.

Properties ​

filepath ​
ts
filepath: string;

Where the response belongs on disk. HTML lands at <pathname>/index.html so a static host serves it for the original path; everything else keeps its own path.

pathname ​
ts
pathname: string;

The path that was requested, exactly as it was queued.

response ​
ts
response: Response;

The router's response. Unconsumed: the body is still readable.

CrawlTarget ​

What crawl needs from a router. A createRouter() router satisfies it, and so does anything else that answers a Request — a built server bundle's default export, a worker-style { fetch } object, a test double.

Methods ​

fetch() ​
ts
fetch(request): Response | Promise<Response>;
Parameters ​
request ​

Request

Returns ​

Response | Promise<Response>

Functions ​

crawl() ​

ts
function crawl(router, options?): AsyncIterableIterator<CrawlResult>;

Walks an app by dispatching requests straight into its router, yielding each response as it arrives. No socket, no server, no browser: the router's fetch is the whole transport, so this runs anywhere the app itself runs.

Yields in completion order. Every path is fetched at most once. A redirect yields nothing and reports through CrawlOptions.onRedirect; any other non-2xx response aborts the crawl.

Parameters ​

router ​

CrawlTarget

The router to crawl.

options? ​

CrawlOptions = {}

Crawl options.

Returns ​

AsyncIterableIterator<CrawlResult>

An async iterator of results, one per fetched path.

staticPaths() ​

ts
function staticPaths(routes): string[];

Collects every path in a route map that can be requested without params: the Remix 3 answer to "what pages does this app have", and the input a prerender pass needs before it knows any dynamic values.

A route qualifies when it answers GET (or any method) and its pattern declares no variables or wildcards. /blog is a static path; /blog/:slug is not, because its values live outside the route map. Routes constrained to a protocol or hostname are skipped too, since their href is not a path.

Results are deduplicated and sorted, so a build that prerenders them lists its output the same way every time.

Parameters ​

routes ​

RouteMap

The route map, usually the one the app's router is built from.

Returns ​

string[]

The static paths, sorted.

© 2026 Pitlane contributors.