---
title: "CrawlOptions"
description: "What [crawl](https://pitlane.tools/package/crawler/function/crawl.md) fetches and how: the starting paths, whether to follow links and queue assets, how many requests run at once, and hooks for `nofollow` pages and redirects. Every option has a default, so `crawl(router)` walks everything reachable from `/`."
url: "https://pitlane.tools/package/crawler/interface/CrawlOptions"
---

# CrawlOptions

```ts
import type { CrawlOptions } from "@pitlane/crawler";
```

What [crawl](https://pitlane.tools/package/crawler/function/crawl.md) fetches and how: the starting paths, whether to follow links and queue assets, how many requests run at once, and hooks for `nofollow` pages and redirects. Every option has a default, so `crawl(router)` walks everything reachable from `/`.

## See

[Crawler guide](https://pitlane.tools/guides/crawler.md)

## Properties

### assets?

```ts
optional assets?: boolean;
```

Queue the `<link href>`, `<script src>`, and `<img src>` a page references. Turn it off when something else already emitted those files, as a bundler does.

#### Default

```ts
true
```

---

### concurrency?

```ts
optional concurrency?: number;
```

How many paths to fetch at once.

#### Default

```ts
1
```

---

### ignorePageNofollow?

```ts
optional ignorePageNofollow?: (pathname: string) => boolean;
```

Crawl a page's links even though the page asked robots not to follow them. Receives the page's path.

#### Parameters

##### pathname

`string`

#### Returns

`boolean`

---

### onRedirect?

```ts
optional onRedirect?: (pathname: string, location: string | null) => void;
```

Called for a path that answered with a redirect rather than a document. Nothing is yielded for it: there is no page to write, and the app still answers the path at runtime.

Receives the requested path and the `Location` it pointed at, which is `null` for a redirect that named none.

#### Parameters

##### pathname

`string`

##### location

`string` | `null`

#### Returns

`void`

---

### paths?

```ts
optional paths?: string[];
```

Paths to start from.

#### Default

```ts
["/"]
```

---

### spider?

```ts
optional spider?: boolean;
```

Follow `<a href>` and `<link rel="alternate">` to discover more paths. Turn it off to fetch exactly the paths given.

#### Default

```ts
true
```