# Scraping GitHub (/developer-guides/common-sites/github)

<!-- agent-signals: reading_time_min: 3 · est_tokens: 1389 · updated: 2026-07-30 -->
Related: [Scraping Amazon](/developer-guides/common-sites/amazon.md), [Scraping Etsy](/developer-guides/common-sites/etsy.md), [Scraping Wikipedia](/developer-guides/common-sites/wikipedia.md)

Learn how to use Firecrawl's core features to scrape GitHub repositories, issues, and documentation.

<Info>
  There are two ways to get GitHub data with Firecrawl. You can **scrape, map, or crawl** a specific `github.com` URL (what the rest of this page covers), or you can use the [Research Index](/features/research) GitHub search to semantically discover relevant issues, pull requests, discussions, and READMEs across GitHub without knowing the exact URL.
</Info>

## Setup [#setup]

```bash
npm install firecrawl zod
```

## Scrape with JSON Mode [#scrape-with-json-mode]

Extract structured data from repositories using Zod schemas.

```typescript
import { Firecrawl } from 'firecrawl';
import { z } from 'zod';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const result = await firecrawl.scrape('https://github.com/firecrawl/firecrawl', {
    formats: [{
        type: 'json',
        schema: z.object({
            name: z.string(),
            description: z.string(),
            stars: z.number(),
            forks: z.number(),
            language: z.string(),
            topics: z.array(z.string())
        })
    }]
});

console.log(result.json);
```

## Search [#search]

Find repositories, issues, or documentation on GitHub.

```typescript
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const searchResult = await firecrawl.search('machine learning site:github.com', {
    limit: 10,
    sources: [{ type: 'web' }], // { type: 'news' }, { type: 'images' }
    scrapeOptions: {
        formats: ['markdown']
    }
});

console.log(searchResult);
```

## Search GitHub with the Research Index [#search-github-with-the-research-index]

Firecrawl's [Research Index](/features/research) includes a GitHub index that semantically searches indexed public GitHub issues, pull requests, discussions, and repository READMEs. It's useful for finding implementation details and engineering prior art when you don't know the exact repository or URL.

```typescript
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const results = await firecrawl.research.searchGithub(
    'how to handle rate limit retries with exponential backoff',
    { k: 10 }
);

console.log(results);
```

You can also reach the same GitHub index via the general [`/search`](/features/search) endpoint with the `github` category.

```typescript
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const searchResult = await firecrawl.search('exponential backoff retry', {
    categories: ['github'],
    limit: 10
});

console.log(searchResult);
```

See the [Research Index guide](/features/research) for Python, cURL, CLI, and MCP usage.

## Scrape [#scrape]

Scrape a single GitHub page - repository, issue, or file.

```typescript
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const result = await firecrawl.scrape('https://github.com/firecrawl/firecrawl', {
    formats: ['markdown'] // i.e. html, links, etc.
});

console.log(result);
```

## Map [#map]

Discover all available URLs in a repository or documentation site. Note: Map returns URLs only, without content.

```typescript
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const mapResult = await firecrawl.map('https://github.com/vercel/next.js/tree/canary/docs');

console.log(mapResult.links);
// Returns array of URLs without content
```

## Crawl [#crawl]

Crawl multiple pages from a repository or documentation.

```typescript
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const crawlResult = await firecrawl.crawl('https://github.com/facebook/react/wiki', {
    limit: 10,
    scrapeOptions: {
        formats: ['markdown']
    }
});

console.log(crawlResult.data);
```

## Batch Scrape [#batch-scrape]

Scrape multiple GitHub URLs simultaneously.

```typescript
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

// Wait for completion
const job = await firecrawl.batchScrape([
    'https://github.com/vercel/next.js',
    'https://github.com/facebook/react',
    'https://github.com/microsoft/typescript'],
    {
        options: {
            formats: ['markdown']
        },
        pollInterval: 2,
        timeout: 120
    }
);


console.log(job.status, job.completed, job.total);

console.log(job);
```

## Batch Scrape with JSON Mode [#batch-scrape-with-json-mode]

Extract structured data from multiple repositories at once.

```typescript
import { Firecrawl } from 'firecrawl';
import { z } from 'zod';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

// Wait for completion
const job = await firecrawl.batchScrape([
    'https://github.com/vercel/next.js',
    'https://github.com/facebook/react'],
    {
        options: {
            formats: [{
                type: 'json',
                schema: z.object({
                    name: z.string(),
                    description: z.string(),
                    stars: z.number(),
                    language: z.string()
                })
            }]
        },
        pollInterval: 2,
        timeout: 120
    }
);


console.log(job.status, job.completed, job.total);

console.log(job);
```