# Crawl (/features/crawl)

<!-- agent-signals: reading_time_min: 14 · est_tokens: 8188 · updated: 2026-07-30 -->
Related: [Search](/features/search.md), [Search Highlights](/features/search-highlights.md), [Research Index](/features/research.md), [Scrape](/features/scrape.md), [Faster Scraping](/features/fast-scraping.md), [Batch Scrape](/features/batch-scrape.md)

Crawl submits a URL to Firecrawl and recursively discovers and scrapes every reachable subpage. It handles sitemaps, JavaScript rendering, and rate limits automatically, returning clean markdown or structured data for each page.

* Discovers pages via sitemap and recursive link traversal
* Supports path filtering, depth limits, and subdomain/external link control
* Returns results via polling, WebSocket, or webhook

<Card title="Try it in the Playground" icon="<svg xmlns=&#x22;http://www.w3.org/2000/svg&#x22; viewBox=&#x22;0 0 24 24&#x22; fill=&#x22;none&#x22;><path d=&#x22;M18.8906 12.846C18.5371 14.189 16.8667 15.138 13.5257 17.0361C10.296 18.8709 8.6812 19.7884 7.37983 19.4196C6.8418 19.2671 6.35159 18.9776 5.95624 18.5787C5 17.6139 5 15.7426 5 12C5 8.2574 5 6.3861 5.95624 5.42132C6.35159 5.02245 6.8418 4.73288 7.37983 4.58042C8.6812 4.21165 10.296 5.12907 13.5257 6.96393C16.8667 8.86197 18.5371 9.811 18.8906 11.154C19.0365 11.7084 19.0365 12.2916 18.8906 12.846Z&#x22; stroke=&#x22;currentColor&#x22; stroke-linejoin=&#x22;round&#x22; stroke-width=&#x22;1.5&#x22;/></svg>" href="https://www.firecrawl.dev/playground?endpoint=crawl">
  Test crawling in the interactive playground — no code required.
</Card>

## Installation [#installation]

<CodeGroup>
  <CodeBlockTabs defaultValue="Python" groupId="cli+node+python">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="Python">
        Python
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Node">
        Node
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="CLI">
        CLI
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="Python">
      ```python  
      # pip install firecrawl-py

      from firecrawl import Firecrawl

      firecrawl = Firecrawl(
        # No API key needed to get started — add one for higher rate limits:
        # api_key="fc-YOUR-API-KEY",
      )
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Node">
      ```js  
      // npm install firecrawl

      import { Firecrawl } from 'firecrawl';

      const firecrawl = new Firecrawl({
        // No API key needed to get started — add one for higher rate limits:
        // apiKey: "fc-YOUR-API-KEY",
      });
      ```
    </CodeBlockTab>

    <CodeBlockTab value="CLI">
      ```bash  
      # Install globally with npm
      npm install -g firecrawl

      # Authenticate (one-time setup)
      firecrawl login
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

## Basic usage [#basic-usage]

Submit a crawl job by calling `POST /v2/crawl` with a starting URL. The endpoint returns a job ID that you use to poll for results.

<CodeGroup>
  <CodeBlockTabs defaultValue="Python" groupId="cli+curl+node+python">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="Python">
        Python
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Node">
        Node
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="cURL">
        cURL
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="CLI">
        CLI
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="Python">
      ```python  
      from firecrawl import Firecrawl

      firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")

      docs = firecrawl.crawl(url="https://docs.firecrawl.dev", limit=10)
      print(docs)
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Node">
      ```js  
      import { Firecrawl } from 'firecrawl';

      const firecrawl = new Firecrawl({ apiKey: "fc-YOUR-API-KEY" });

      const docs = await firecrawl.crawl('https://docs.firecrawl.dev', { limit: 10 });
      console.log(docs);
      ```
    </CodeBlockTab>

    <CodeBlockTab value="cURL">
      ```bash  
      curl -s -X POST "https://api.firecrawl.dev/v2/crawl" \
        -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
          "url": "https://docs.firecrawl.dev",
          "limit": 10
        }'
      ```
    </CodeBlockTab>

    <CodeBlockTab value="CLI">
      ```bash  
      # Start a crawl job (returns job ID)
      firecrawl crawl https://firecrawl.dev

      # Wait for completion with progress
      firecrawl crawl https://firecrawl.dev --wait --progress --limit 100
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

<Info>
  Each page crawled consumes 1 credit. The default crawl `limit` is 10,000 pages. Before starting, the crawl endpoint checks that your remaining credits can cover the `limit` — if not, it returns a &#x2A;*402 (Payment Required)** error. Set a lower `limit` to match your intended crawl size (e.g. `limit: 100`) to avoid this. Additional credits apply for certain options: JSON mode costs 4 additional credits per page, enhanced proxy costs 4 additional credits per page, and PDF parsing costs 1 credit per PDF page.
</Info>

### Scrape options [#scrape-options]

All options from the [Scrape endpoint](/api-reference/endpoint/scrape) are available in crawl via `scrapeOptions` (JS) / `scrape_options` (Python). These apply to every page the crawler scrapes, including formats, proxy, caching, actions, location, and tags.

<CodeGroup>
  <CodeBlockTabs defaultValue="Python" groupId="node+python">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="Python">
        Python
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Node">
        Node
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="Python">
      ```python  
      from firecrawl import Firecrawl

      firecrawl = Firecrawl(api_key='fc-YOUR_API_KEY')

      # Crawl with scrape options
      response = firecrawl.crawl('https://example.com',
          limit=100,
          scrape_options={
              'formats': [
                  'markdown',
                  { 'type': 'json', 'schema': { 'type': 'object', 'properties': { 'title': { 'type': 'string' } } } }
              ],
              'proxy': 'auto',
              'max_age': 600000,
              'only_main_content': True
          }
      )
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Node">
      ```js  
      import { Firecrawl } from 'firecrawl';

      const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR_API_KEY' });

      // Crawl with scrape options
      const crawlResponse = await firecrawl.crawl('https://example.com', {
        limit: 100,
        scrapeOptions: {
          formats: [
            'markdown',
            {
              type: 'json',
              schema: { type: 'object', properties: { title: { type: 'string' } } },
            },
          ],
          proxy: 'auto',
          maxAge: 600000,
          onlyMainContent: true,
        },
      });
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

## Checking crawl status [#checking-crawl-status]

Use the job ID to poll for the crawl status and retrieve results.

<CodeGroup>
  <CodeBlockTabs defaultValue="Python" groupId="cli+curl+node+python">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="Python">
        Python
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Node">
        Node
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="cURL">
        cURL
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="CLI">
        CLI
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="Python">
      ```python  
      status = firecrawl.get_crawl_status("<crawl-id>")
      print(status)
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Node">
      ```js  
      const status = await firecrawl.getCrawlStatus("<crawl-id>");
      console.log(status);
      ```
    </CodeBlockTab>

    <CodeBlockTab value="cURL">
      ```bash  
      # After starting a crawl, poll status by jobId
      curl -s -X GET "https://api.firecrawl.dev/v2/crawl/<jobId>" \
        -H "Authorization: Bearer $FIRECRAWL_API_KEY"
      ```
    </CodeBlockTab>

    <CodeBlockTab value="CLI">
      ```bash  
      # Check crawl status using job ID
      firecrawl crawl <job-id>
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

<Note>
  Job results are available via the API for 24 hours after completion. After this period, you can still view your crawl history and results in the [activity logs](https://www.firecrawl.dev/app/logs).
</Note>

<Note>
  Pages in the crawl results `data` array are pages that Firecrawl successfully scraped, even if the target site returned an HTTP error like 404. The `metadata.statusCode` field shows the HTTP status code from the target site. To retrieve pages that Firecrawl itself failed to scrape (e.g. network errors, timeouts, or robots.txt blocks), use the dedicated [Get Crawl Errors](/api-reference/endpoint/crawl-get-errors) endpoint (`GET /crawl/{id}/errors`).
</Note>

### Response handling [#response-handling]

The response varies based on the crawl's status. For incomplete or large responses exceeding 10MB, a `next` URL parameter is provided. You must request this URL to retrieve the next 10MB of data. If the `next` parameter is absent, it indicates the end of the crawl data.

<Info>
  The `skip` and `next` parameters are only relevant when hitting the API directly.
  If you're using the SDK, pagination is handled automatically and all
  results are returned at once.
</Info>

<CodeGroup>
  <CodeBlockTabs defaultValue="Scraping" groupId="completed+scraping">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="Scraping">
        Scraping
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Completed">
        Completed
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="Scraping">
      ```json  
      {
        "status": "scraping",
        "total": 36,
        "completed": 10,
        "creditsUsed": 10,
        "expiresAt": "2024-00-00T00:00:00.000Z",
        "next": "https://api.firecrawl.dev/v2/crawl/123-456-789?skip=10",
        "data": [
          {
            "markdown": "[Firecrawl Docs home page![light logo](https://mintlify.s3-us-west-1.amazonaws.com/firecrawl/logo/light.svg)!...",
            "html": "<!DOCTYPE html><html lang=\"en\" class=\"js-focus-visible lg:[--scroll-mt:9.5rem]\" data-js-focus-visible=\"\">...",
            "metadata": {
              "title": "Build a 'Chat with website' using Groq Llama 3 | Firecrawl",
              "language": "en",
              "sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3",
              "description": "Learn how to use Firecrawl, Groq Llama 3, and Langchain to build a 'Chat with your website' bot.",
              "ogLocaleAlternate": [],
              "statusCode": 200
            }
          },
          ...
        ]
      }
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Completed">
      ```json  
      {
        "status": "completed",
        "total": 36,
        "completed": 36,
        "creditsUsed": 36,
        "expiresAt": "2024-00-00T00:00:00.000Z",
        "next": "https://api.firecrawl.dev/v2/crawl/123-456-789?skip=26",
        "data": [
          {
            "markdown": "[Firecrawl Docs home page![light logo](https://mintlify.s3-us-west-1.amazonaws.com/firecrawl/logo/light.svg)!...",
            "html": "<!DOCTYPE html><html lang=\"en\" class=\"js-focus-visible lg:[--scroll-mt:9.5rem]\" data-js-focus-visible=\"\">...",
            "metadata": {
              "title": "Build a 'Chat with website' using Groq Llama 3 | Firecrawl",
              "language": "en",
              "sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3",
              "description": "Learn how to use Firecrawl, Groq Llama 3, and Langchain to build a 'Chat with your website' bot.",
              "ogLocaleAlternate": [],
              "statusCode": 200
            }
          },
          ...
        ]
      }
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

## SDK methods [#sdk-methods]

There are two ways to use crawl with the SDK.

### Crawl and wait [#crawl-and-wait]

The `crawl` method waits for the crawl to complete and returns the full response. It handles pagination automatically. This is recommended for most use cases.

<CodeGroup>
  <CodeBlockTabs defaultValue="Python" groupId="node+python">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="Python">
        Python
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Node">
        Node
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="Python">
      ```python  
      from firecrawl import Firecrawl
      from firecrawl.types import ScrapeOptions

      firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")

      # Crawl a website:
      crawl_status = firecrawl.crawl(
        'https://firecrawl.dev', 
        limit=100, 
        scrape_options=ScrapeOptions(formats=['markdown', 'html']),
        poll_interval=30
      )
      print(crawl_status)
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Node">
      ```js  
      import { Firecrawl } from 'firecrawl';

      const firecrawl = new Firecrawl({apiKey: "fc-YOUR_API_KEY"});

      const crawlResponse = await firecrawl.crawl('https://firecrawl.dev', {
        limit: 100,
        scrapeOptions: {
          formats: ['markdown', 'html'],
        }
      })

      console.log(crawlResponse)
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

The response includes the crawl status and all scraped data:

<CodeGroup>
  <CodeBlockTabs defaultValue="Python" groupId="node+python">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="Python">
        Python
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Node">
        Node
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="Python">
      ```bash  
      success=True
      status='completed'
      completed=100
      total=100
      creditsUsed=100
      expiresAt=datetime.datetime(2025, 4, 23, 19, 21, 17, tzinfo=TzInfo(UTC))
      next=None
      data=[
        Document(
          markdown='[Day 7 - Launch Week III.Integrations DayApril 14th to 20th](...',
          metadata={
            'title': '15 Python Web Scraping Projects: From Beginner to Advanced',
            ...
            'scrapeId': '97dcf796-c09b-43c9-b4f7-868a7a5af722',
            'sourceURL': 'https://www.firecrawl.dev/blog/python-web-scraping-projects',
            'url': 'https://www.firecrawl.dev/blog/python-web-scraping-projects',
            'statusCode': 200
          }
        ),
        ...
      ]
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Node">
      ```js  
      {
        success: true,
        status: "completed",
        completed: 100,
        total: 100,
        creditsUsed: 100,
        expiresAt: "2025-04-23T19:28:45.000Z",
        data: [
          {
            markdown: "[Day 7 - Launch Week III.Integrations DayApril ...",
            html: `<!DOCTYPE html><html lang="en" class="light" style="color...`,
            metadata: [Object],
          },
          ...
        ]
      }
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

### Start and check later [#start-and-check-later]

The `startCrawl` / `start_crawl` method returns immediately with a crawl ID. You then poll for status manually. This is useful for long-running crawls or custom polling logic.

<CodeGroup>
  <CodeBlockTabs defaultValue="Python" groupId="cli+curl+node+python">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="Python">
        Python
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Node">
        Node
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="cURL">
        cURL
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="CLI">
        CLI
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="Python">
      ```python  
      from firecrawl import Firecrawl

      firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")

      job = firecrawl.start_crawl(url="https://docs.firecrawl.dev", limit=10)
      print(job)

      # Check the status of the crawl
      status = firecrawl.get_crawl_status(job.id)
      print(status)
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Node">
      ```js  
      import { Firecrawl } from 'firecrawl';

      const firecrawl = new Firecrawl({ apiKey: "fc-YOUR-API-KEY" });

      const { id } = await firecrawl.startCrawl('https://docs.firecrawl.dev', { limit: 10 });
      console.log(id);

      // Check the status of the crawl
      const status = await firecrawl.getCrawlStatus(id);
      console.log(status);

      ```
    </CodeBlockTab>

    <CodeBlockTab value="cURL">
      ```bash  
      curl -s -X POST "https://api.firecrawl.dev/v2/crawl" \
        -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
        -H "Content-Type: application/json" \
        -d '{
          "url": "https://docs.firecrawl.dev",
          "limit": 10
        }'
      ```
    </CodeBlockTab>

    <CodeBlockTab value="CLI">
      ```bash  
      # Start crawl (async, returns job ID immediately)
      firecrawl crawl https://firecrawl.dev --limit 100

      # Then check status later
      firecrawl crawl <job-id>
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

The initial response returns the job ID:

```json
{
  "success": true,
  "id": "123-456-789",
  "url": "https://api.firecrawl.dev/v2/crawl/123-456-789"
}
```

## Real-time results with WebSocket [#real-time-results-with-websocket]

The watcher method provides real-time updates as pages are crawled. Start a crawl, then subscribe to events for immediate data processing.

<CodeGroup>
  <CodeBlockTabs defaultValue="Python" groupId="node+python">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="Python">
        Python
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Node">
        Node
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="Python">
      ```python  
      import asyncio
      from firecrawl import AsyncFirecrawl

      async def main():
          firecrawl = AsyncFirecrawl(api_key="fc-YOUR-API-KEY")

          # Start a crawl first
          started = await firecrawl.start_crawl("https://firecrawl.dev", limit=5)

          # Watch updates (snapshots) until terminal status
          async for snapshot in firecrawl.watcher(started.id, kind="crawl", poll_interval=2, timeout=120):
              if snapshot.status == "completed":
                  print("DONE", snapshot.status)
                  for doc in snapshot.data:
                      print("DOC", doc.metadata.source_url if doc.metadata else None)
              elif snapshot.status == "failed":
                  print("ERR", snapshot.status)
              else:
                  print("STATUS", snapshot.status, snapshot.completed, "/", snapshot.total)

      asyncio.run(main())
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Node">
      ```js  
      import { Firecrawl } from 'firecrawl';

      const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR-API-KEY' });

      // Start a crawl and then watch it
      const { id } = await firecrawl.startCrawl('https://mendable.ai', {
        excludePaths: ['blog/*'],
        limit: 5,
      });

      const watcher = firecrawl.watcher(id, { kind: 'crawl', pollInterval: 2, timeout: 120 });

      watcher.on('document', (doc) => {
        console.log('DOC', doc);
      });

      watcher.on('error', (err) => {
        console.error('ERR', err?.error || err);
      });

      watcher.on('done', (state) => {
        console.log('DONE', state.status);
      });

      // Begin watching (WS with HTTP fallback)
      await watcher.start();
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

## Webhooks [#webhooks]

You can configure webhooks to receive real-time notifications as your crawl progresses. This allows you to process pages as they are scraped instead of waiting for the entire crawl to complete.

```bash title="cURL"
curl -X POST https://api.firecrawl.dev/v2/crawl \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://docs.firecrawl.dev",
      "limit": 100,
      "webhook": {
        "url": "https://your-domain.com/webhook",
        "metadata": {
          "any_key": "any_value"
        },
        "events": ["started", "page", "completed"]
      }
    }'
```

### Event types [#event-types]

| Event             | Description                              |
| ----------------- | ---------------------------------------- |
| `crawl.started`   | Fires when the crawl begins              |
| `crawl.page`      | Fires for each page successfully scraped |
| `crawl.completed` | Fires when the crawl finishes            |
| `crawl.failed`    | Fires if the crawl encounters an error   |

### Payload [#payload]

```json
{
  "success": true,
  "type": "crawl.page",
  "id": "crawl-job-id",
  "data": [...], // Page data for 'page' events
  "metadata": {}, // Your custom metadata
  "error": null
}
```

### Verifying webhook signatures [#verifying-webhook-signatures]

Every webhook request from Firecrawl includes an `X-Firecrawl-Signature` header containing an HMAC-SHA256 signature. Always verify this signature to ensure the webhook is authentic and has not been tampered with.

1. Get your webhook secret from the [Advanced tab](https://www.firecrawl.dev/app/settings?tab=advanced) of your account settings
2. Extract the signature from the `X-Firecrawl-Signature` header
3. Compute HMAC-SHA256 of the raw request body using your secret
4. Compare with the signature header using a timing-safe function

<Warning>
  Never process a webhook without verifying its signature first. The `X-Firecrawl-Signature` header contains the signature in the format: `sha256=abc123def456...`
</Warning>

For complete implementation examples in JavaScript and Python, see the [Webhook Security documentation](/webhooks/security). For comprehensive webhook documentation including detailed event payloads, payload structure, advanced configuration, and troubleshooting, see the [Webhooks documentation](/webhooks/overview).

## Configuration reference [#configuration-reference]

The full set of parameters available when submitting a crawl job:

| Parameter               | Type       | Default     | Description                                                                                                                                                                                                                                                                                                                                                          |
| ----------------------- | ---------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url`                   | `string`   | (required)  | The starting URL to crawl from                                                                                                                                                                                                                                                                                                                                       |
| `limit`                 | `integer`  | `10000`     | Maximum number of pages to crawl                                                                                                                                                                                                                                                                                                                                     |
| `maxDiscoveryDepth`     | `integer`  | (none)      | Maximum depth from the root URL based on link-discovery hops, not the number of `/` segments in the URL. Each time a new URL is found on a page, it is assigned a depth one higher than the page it was discovered on. The root site and sitemapped pages have a discovery depth of 0. Pages at the max depth are still scraped, but links on them are not followed. |
| `includePaths`          | `string[]` | (none)      | URL pathname regex patterns to include. Only matching paths are crawled.                                                                                                                                                                                                                                                                                             |
| `excludePaths`          | `string[]` | (none)      | URL pathname regex patterns to exclude from the crawl                                                                                                                                                                                                                                                                                                                |
| `regexOnFullURL`        | `boolean`  | `false`     | Match `includePaths`/`excludePaths` against the full URL (including query parameters) instead of just the pathname                                                                                                                                                                                                                                                   |
| `crawlEntireDomain`     | `boolean`  | `false`     | Follow internal links to sibling or parent URLs, not just child paths                                                                                                                                                                                                                                                                                                |
| `allowSubdomains`       | `boolean`  | `false`     | Follow links to subdomains of the main domain                                                                                                                                                                                                                                                                                                                        |
| `allowExternalLinks`    | `boolean`  | `false`     | Follow links to external websites                                                                                                                                                                                                                                                                                                                                    |
| `sitemap`               | `string`   | `"include"` | Sitemap handling: `"include"` (default), `"skip"`, or `"only"`                                                                                                                                                                                                                                                                                                       |
| `ignoreQueryParameters` | `boolean`  | `false`     | Avoid re-scraping the same path with different query parameters                                                                                                                                                                                                                                                                                                      |
| `ignoreRobotsTxt`       | `boolean`  | `false`     | Ignore the website's robots.txt rules. **Enterprise only** — contact [support@firecrawl.com](mailto:support@firecrawl.com) to enable.                                                                                                                                                                                                                                |
| `robotsUserAgent`       | `string`   | (none)      | Custom User-Agent string for robots.txt evaluation. When set, robots.txt is fetched with this User-Agent and rules are matched against it instead of the default. **Enterprise only** — contact [support@firecrawl.com](mailto:support@firecrawl.com) to enable.                                                                                                     |
| `delay`                 | `number`   | (none)      | Delay in seconds between scrapes to respect rate limits. Setting this forces concurrency to 1.                                                                                                                                                                                                                                                                       |
| `maxConcurrency`        | `integer`  | (none)      | Maximum concurrent scrapes. Defaults to your team's concurrency limit.                                                                                                                                                                                                                                                                                               |
| `scrapeOptions`         | `object`   | (none)      | Options applied to every scraped page (formats, proxy, caching, actions, etc.)                                                                                                                                                                                                                                                                                       |
| `webhook`               | `object`   | (none)      | Webhook configuration for real-time notifications                                                                                                                                                                                                                                                                                                                    |
| `prompt`                | `string`   | (none)      | Natural language prompt to generate crawl options. Explicitly set parameters override generated equivalents.                                                                                                                                                                                                                                                         |

## Important details [#important-details]

<Warning>
  By default, crawl ignores sublinks that are not children of the URL you provide. For example, `website.com/other-parent/blog-1` would not be returned if you crawled `website.com/blogs/`. Use the `crawlEntireDomain` parameter to include sibling and parent paths. To crawl subdomains like `blog.website.com` when crawling `website.com`, use the `allowSubdomains` parameter.
</Warning>

* **Sitemap discovery**: By default, the crawler includes the website's sitemap to discover URLs (`sitemap: "include"`). If you set `sitemap: "skip"`, only pages reachable through HTML links from the root URL are found. Assets like PDFs or deeply nested pages listed in the sitemap but not directly linked from HTML will be missed. For maximum coverage, keep the default setting.
* **Credit usage**: Each page crawled costs 1 credit. JSON mode adds 4 credits per page, enhanced proxy adds 4 credits per page, and PDF parsing costs 1 credit per PDF page.
* **Result expiration**: Job results are available via the API for 24 hours after completion. After that, view results in the [activity logs](https://www.firecrawl.dev/app/logs).
* **Crawl errors**: The `data` array contains pages Firecrawl successfully scraped. Use the [Get Crawl Errors](/api-reference/endpoint/crawl-get-errors) endpoint to retrieve pages that failed due to network errors, timeouts, or robots.txt blocks.
* **Non-deterministic results**: Crawl results may vary between runs of the same configuration. Pages are scraped concurrently, so the order in which links are discovered depends on network timing and which pages finish loading first. This means different branches of a site may be explored to different extents near the depth boundary, especially at higher `maxDiscoveryDepth` values. To get more deterministic results, set `maxConcurrency` to `1` or use `sitemap: "only"` if the site has a comprehensive sitemap.

> Are you an AI agent that needs a Firecrawl API key? See [firecrawl.dev/agent-onboarding/SKILL.md](https://www.firecrawl.dev/agent-onboarding/SKILL.md) for automated onboarding instructions.