Building an AI Research Assistant with Firecrawl and AI SDK

Create a New Next.js Project

Start by creating a fresh Next.js application and navigating into the project directory:

npx create-next-app@latest ai-sdk-firecrawl && cd ai-sdk-firecrawl

When prompted, select the following options:

• TypeScript: Yes
• ESLint: Yes
• Tailwind CSS: Yes
• App Router: Yes
• Use src/ directory: No
• Import alias: Yes (@/*)

Install Dependencies

Install AI SDK Packages

The AI SDK is a TypeScript toolkit that provides a unified API for working with different LLM providers:

npm i ai @ai-sdk/react zod

These packages provide:

• ai: Core SDK with streaming, tool calling, and response handling
• @ai-sdk/react: React hooks like useChat for building chat interfaces
• zod: Schema validation for tool inputs

Learn more at ai-sdk.dev/docs.

Install AI Elements

AI Elements provides pre-built UI components for AI applications. Run the following command to scaffold all the necessary components:

npx ai-elements@latest

This sets up AI Elements in your project, including conversation components, message displays, prompt inputs, and tool call visualizations.

Documentation: ai-sdk.dev/elements/overview.

Install OpenAI Provider

Install the OpenAI provider to connect with OpenAI’s models:

npm install @ai-sdk/openai

Build the Frontend Chat Interface

Create the main page at app/page.tsx and copy the code from the Code tab below. This will be the chat interface where users interact with the AI assistant.

Understanding the Frontend

The frontend uses AI Elements components to provide a complete chat interface:

Key Features:

• Conversation Display: The Conversation component automatically handles message scrolling and display
• Message Rendering: Each message part is rendered based on its type (text, reasoning, tool calls)
• Tool Visualization: Tool calls are displayed with collapsible sections showing inputs and outputs
• Interactive Controls: Users can toggle web search, select models, and attach files
• Message Actions: Copy and retry actions for assistant messages

Add Markdown Rendering Support

To ensure the markdown from the LLM is correctly rendered, add the following import to your app/globals.css file:

@source "../node_modules/streamdown/dist/index.js";

This imports the necessary styles for rendering markdown content in the message responses.

Build the Basic API Route

Create the chat API endpoint at app/api/chat/route.ts. This route will handle incoming messages and stream responses from the AI.

import { streamText, UIMessage, convertToModelMessages } from "ai";
import { createOpenAI } from "@ai-sdk/openai";

const openai = createOpenAI({
  apiKey: process.env.OPENAI_API_KEY!,
});

// Allow streaming responses up to 5 minutes
export const maxDuration = 300;

export async function POST(req: Request) {
  const {
    messages,
    model,
    webSearch,
  }: {
    messages: UIMessage[];
    model: string;
    webSearch: boolean;
  } = await req.json();

  const result = streamText({
    model: openai(model),
    messages: convertToModelMessages(messages),
    system:
      "You are a helpful assistant that can answer questions and help with tasks.",
  });

  // send sources and reasoning back to the client
  return result.toUIMessageStreamResponse({
    sendSources: true,
    sendReasoning: true,
  });
}

This basic route:

• Receives messages from the frontend
• Uses the OpenAI model selected by the user
• Streams responses back to the client
• Doesn’t include tools yet - we’ll add those next

Configure Environment Variables

Create a .env.local file in your project root:

touch .env.local

Add your OpenAI API key:

OPENAI_API_KEY=sk-your-openai-api-key

The OPENAI_API_KEY is required for the AI model to function.

Test the Basic Chat

Now you can test the AI SDK chatbot without Firecrawl integration. Start the development server:

npm run dev

Open localhost:3000 in your browser and test the basic chat functionality. The assistant should respond to messages, but won’t have web scraping or search capabilities yet.

Add Firecrawl Tools

Now let’s enhance the assistant with web scraping and search capabilities using Firecrawl.

Install Firecrawl SDK

Firecrawl converts websites into LLM-ready formats with scraping and search capabilities:

npm i firecrawl

Create the Tools File

Create a lib folder and add a tools.ts file inside it:

mkdir lib && touch lib/tools.ts

Add the following code to define the web scraping and search tools:

import { Firecrawl } from "firecrawl";
import { tool } from "ai";
import { z } from "zod";

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

export const scrapeWebsiteTool = tool({
  description: 'Scrape content from any website URL',
  inputSchema: z.object({
    url: z.string().url().describe('The URL to scrape')
  }),
  execute: async ({ url }) => {
    console.log('Scraping:', url);
    const result = await firecrawl.scrape(url, {
      formats: ['markdown'],
      onlyMainContent: true,
      timeout: 30000
    });
    console.log('Scraped content preview:', result.markdown?.slice(0, 200) + '...');
    return { content: result.markdown };
  }
});

export const searchWebTool = tool({
  description: 'Search the web using Firecrawl',
  inputSchema: z.object({
    query: z.string().describe('The search query'),
    limit: z.number().optional().describe('Number of results'),
    location: z.string().optional().describe('Location for localized results'),
    tbs: z.string().optional().describe('Time filter (qdr:h, qdr:d, qdr:w, qdr:m, qdr:y)'),
    sources: z.array(z.enum(['web', 'news', 'images'])).optional().describe('Result types'),
    categories: z.array(z.enum(['github', 'research', 'pdf'])).optional().describe('Filter categories'),
  }),
  execute: async ({ query, limit, location, tbs, sources, categories }) => {
    console.log('Searching:', query);
    const response = await firecrawl.search(query, {
      ...(limit && { limit }),
      ...(location && { location }),
      ...(tbs && { tbs }),
      ...(sources && { sources }),
      ...(categories && { categories }),
    }) as { web?: Array<{ title?: string; url?: string; description?: string }> };

    const results = (response.web || []).map((item) => ({
      title: item.title || item.url || 'Untitled',
      url: item.url || '',
      description: item.description || '',
    }));

    console.log('Search results:', results.length);
    return { results };
  },
});

Understanding the Tools

Scrape Website Tool:

• Accepts a URL as input (validated by Zod schema)
• Uses Firecrawl’s scrape method to fetch the page as markdown
• Extracts only the main content to reduce token usage
• Returns the scraped content for the AI to analyze

Search Web Tool:

• Accepts a search query with optional filters
• Uses Firecrawl’s search method to find relevant web pages
• Supports advanced filters like location, time range, and content categories
• Returns structured results with titles, URLs, and descriptions

Learn more about tools: ai-sdk.dev/docs/foundations/tools.

Update the API Route with Firecrawl Tools

Now update your app/api/chat/route.ts to include the Firecrawl tools we just created.

The key changes from the basic route:

• Import stepCountIs from the AI SDK
• Import the Firecrawl tools from @/lib/tools
• Add the tools object with both scrapeWebsite and searchWeb tools
• Add stopWhen: stepCountIs(5) to limit execution steps
• Set toolChoice to “auto” when web search is enabled, “none” otherwise

Learn more about streamText: ai-sdk.dev/docs/reference/ai-sdk-core/stream-text.

Add Your Firecrawl API Key

Update your .env.local file to include your Firecrawl API key:

OPENAI_API_KEY=sk-your-openai-api-key
FIRECRAWL_API_KEY=fc-your-firecrawl-api-key

Get your Firecrawl API key from firecrawl.dev.

Test the Complete Application

Restart your development server:

npm run dev

Open localhost:3000 and test the enhanced assistant:

1. Toggle the “Search” button to enable web search
2. Ask: “What are the latest features from firecrawl.dev?”
3. Watch as the AI calls the searchWeb or scrapeWebsite tool
4. See the tool execution in the UI with inputs and outputs
5. Read the AI’s analysis based on the scraped data