Prerendering

A page whose content does not change per request does not need a server on the critical path. Render it once, write the HTML to disk, and a static host serves it without waking anything up.

@pitlane/crawler walks the app from a script and hands back each response with the path it belongs at on disk. The whole job is a for await loop and two filesystem calls:

import { crawl } from "@pitlane/crawler";
import * as fs from "node:fs/promises";
import * as path from "node:path";

import router from "./app/entry.server.ts";

for await (let { filepath, response } of crawl(router)) {
    let outputPath = path.join("dist", filepath);
    await fs.mkdir(path.dirname(outputPath), { recursive: true });
    await fs.writeFile(outputPath, new Uint8Array(await response.arrayBuffer()));
}

The crawling guide has the install command for your package manager.

There is no separate rendering path. A Request goes through the same fetch handler production runs, and the response that comes back is the page. Route handlers, middleware, and components behave exactly as they do at runtime, because they are the same code answering the same request.

Both setups write the same thing, real HTML at real paths, and differ only in what drives the rendering. The control at the top of the page chooses which setup this guide describes.

Where the paths come from

crawl(router) starts at / and follows the links it finds there. paths replaces that starting list:

crawl(router, { paths: ["/", "/about", "/pricing"], spider: false });

staticPaths(routes) fills the half of that list the route map already knows. It returns every path the app can serve with no params, which leaves the per-slug half as the only part you write:

import { crawl, staticPaths } from "@pitlane/crawler";

import { routes } from "./app/routes.ts";

crawl(router, {
    paths: [...staticPaths(routes), ...slugs.map(slug => `/blog/${slug}`)],
    spider: false,
});

Asking what pages exist covers what qualifies.

concurrency sets how many paths render at once. Rendering is CPU-bound in process, so the gain depends on how much of a render waits on I/O; start at the default of 1 and measure.

Spidering

Spidering turns the path list into a set of starting points. Every rendered page is scanned for links, and those get rendered too.

It is on by default, so crawl(router) with nothing else already means "everything reachable". Pass spider: false to fetch exactly the paths you gave.

One starting path can be the whole list for a site whose pages all link to each other. Anything reachable gets built, including the page you forgot to list.

Crawling stops where a crawler should stop: rel="nofollow" links, <meta name="robots" content="nofollow"> pages, other origins, and non-navigable hrefs like mailto:.

ignorePageNofollow is the escape hatch for a page whose nofollow is aimed at search engines rather than at you, such as a versioned docs tree that should not be indexed but does need to be built. See where a crawl stops.

Output on disk

Each page becomes an index.html under its own path, so a static host serves it back for the original URL.

filepath is where each response belongs, ready to join onto an output directory. Everything that is not HTML keeps its own path instead.

Assets come along too, which is what a site with no bundler needs: the <link href>, <script src>, and <img src> a page references are fetched and written beside it. Pass assets: false when something else already emitted those files.

Paths that redirect

A redirect is not a page, so nothing is written for one. A route answering 302 is an ordinary thing to find in a route map: a / pointing at the real landing path, or a URL that moved.

Nothing is yielded for it, so the loop writes nothing. onRedirect is called with the path and the Location it pointed at, for a script that wants to report what it skipped.

The app still answers that path at runtime, which is the behavior the redirect was for. Nothing about it changes.

When spidering, a redirect is followed rather than skipped, because following links is what spidering is. A crawl seeded at a / that points elsewhere still reaches the site instead of stopping at the door.

Any other failing response is a real failure and stops the walk, naming the path it came from:

Error: Crawl failed: 404 Not Found (/blog/renamed-post)

A listed path that 404s is a stale prerender list, and a spidered one is a dead internal link. Both are worth knowing about before a deploy rather than after.

Serving the output

That output directory is the whole site, documents and assets both. Point a static host at it.

Data that goes stale

A prerendered page is frozen at the moment it rendered. That is the point, and it is also the constraint: a page showing anything that changes between deploys should not be in the list.

There is no revalidation mechanism here, and no incremental regeneration. Render again and redeploy, or leave the path out and let the server answer it.