Firecrawl Agent

"AI That Navigates Websites Like a Human, Extracts Data Like a Pro" by John Rippy | johnrippy.link

🏆 2025 Zapier Automation Hero of the Year — Project Phoenix: A 95-step AI sales pipeline cutting development time by 50%. Read more →

---

Stop Writing Custom Scrapers for Every Website

You're currently doing: Building different scrapers for each website, handling pagination manually, dealing with JavaScript-rendered content, writing complex CSS selectors. What if you could just describe what you want? Firecrawl Agent uses Firecrawl's autonomous Agent API to navigate websites intelligently and extract exactly what you need:

Natural Language Goals - Describe extraction in plain English
Autonomous Navigation - AI decides which links to click
Smart Pagination - Automatically handles "Load More" and infinite scroll
JavaScript Rendering - Full browser rendering for SPAs
Backtracking - Recovers from dead ends automatically

Powered by Firecrawl's Agent API - the same technology behind production web scraping systems.

---

How It Works

URL + Natural Language Goal
↓
┌─────────────┐
│  GOAL       │ Parse user objective
│  ANALYSIS   │ Identify data requirements
└─────────────┘
↓
┌─────────────┐
│  STRATEGIC  │ Find likely paths to data
│  NAVIGATION │ Click links, fill forms
└─────────────┘
↓
┌─────────────┐
│  CONTENT    │ Render JavaScript
│  EXTRACTION │ Scrape clean content
└─────────────┘
↓
┌─────────────┐
│  DATA       │ Apply optional schema
│  STRUCTURING│ Format as JSON
└─────────────┘
↓
┌─────────────┐
│  VALIDATION │ Verify data meets goal
│             │ Backtrack if needed
└─────────────┘
↓
Clean JSON Output

---

Use Cases

1. Competitive Pricing Research

{
"url": "https://competitor.com",
"objective": "Find all pricing plans, their monthly and annual costs, and included features"
}

Returns:

{
"plans": [
{
"name": "Starter",
"monthlyPrice": 29,
"annualPrice": 290,
"features": ["5 users", "10GB storage", "Email support"]
},
{
"name": "Pro",
"monthlyPrice": 99,
"annualPrice": 990,
"features": ["Unlimited users", "100GB storage", "Priority support"]
}
]
}

2. Job Listings Extraction

{
"url": "https://company.com/careers",
"objective": "Extract all software engineering job openings with title, location, and requirements"
}

3. Product Catalog Scraping

{
"url": "https://store.com/products",
"objective": "Get all products with name, price, description, and availability",
"schema": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": { "type": "string" },
"price": { "type": "number" },
"inStock": { "type": "boolean" }
}
}
}
}

4. Company Research

{
"url": "https://company.com",
"objective": "Find the leadership team, their titles, and LinkedIn profiles"
}

5. Event/Conference Extraction

{
"url": "https://conference.com",
"objective": "Extract the conference schedule with session titles, speakers, times, and room locations"
}

6. Documentation Scraping

{
"url": "https://docs.example.com",
"objective": "Extract all API endpoints with their methods, parameters, and descriptions"
}

---

Input Parameters

*Not required in demo mode

---

Output Format

{
"success": true,
"url": "https://example.com",
"objective": "Find pricing plans",
"data": {
"plans": [
{
"name": "Starter",
"price": 29,
"features": ["5 users", "10GB storage"]
},
{
"name": "Pro",
"price": 99,
"features": ["Unlimited users", "100GB storage", "Priority support"]
}
]
},
"stepsUsed": 4,
"pagesVisited": [
"https://example.com",
"https://example.com/pricing"
],
"extractedAt": "2024-12-23T10:30:00Z"
}

---

Pricing

BYOK (Bring Your Own Key)

Cost Per Extraction

Cost Comparison

No development time. No maintenance. Pay per use.

---

API Integration

Using the Apify API

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('localhowl/firecrawl-agent').call({
url: 'https://competitor.com/pricing',
objective: 'Extract all pricing plans with features and costs',
firecrawlApiKey: 'fc-xxx'
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].data);

Using cURL

curl -X POST "https://api.apify.com/v2/acts/localhowl~firecrawl-agent/runs?token=YOUR_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/pricing",
"objective": "Find all pricing plans with their features",
"firecrawlApiKey": "fc-xxx"
}'

---

Demo Mode

Test the actor without an API key:

{
"demoMode": true,
"url": "https://example-saas.com",
"objective": "Find the pricing information"
}

Returns sample pricing data extraction.

---

Firecrawl Agent vs Fire-Enrich

Use Firecrawl Agent when you need to extract arbitrary data from any website. Use Fire-Enrich when you specifically need to enrich company/lead information.

---

Perfect For

Sales Teams

Extract competitor pricing
Gather prospect information
Build target account lists

Product Managers

Competitive analysis
Feature comparison research
Market research

Developers

API documentation scraping
Data migration projects
Integration research

Researchers

Content aggregation
Data collection for analysis
Automated research tasks

---

Advanced Features

Schema Enforcement

Force output to match a specific structure:

{
"url": "https://store.com/products",
"objective": "Get all products",
"schema": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": { "type": "string" },
"price": { "type": "number" },
"sku": { "type": "string" },
"inStock": { "type": "boolean" }
},
"required": ["name", "price"]
}
}
}

Backtracking

When allowBacktracking: true (default), the agent will:

1. Try the most likely path to find data

2. If unsuccessful, backtrack and try alternative paths

3. Continue until data is found or maxSteps is reached

---

Limitations

Authentication: Cannot access login-protected content
Rate Limits: Respects Firecrawl's API rate limits
Captchas: Cannot bypass CAPTCHA challenges
Heavy JavaScript: Very complex SPAs may require more steps
Dynamic Content: Real-time updating content may vary between runs

---

Support

For issues or feature requests, contact support@localhowl.com

Get your Firecrawl API key at firecrawl.dev

---

Built by John Rippy | johnrippy.link