Web scraping API for LLM apps: Firecrawl and Crawl4AI

By Tracklayer · Updated 2026-11-20

Short answer: self-host a crawling API that fetches pages in headless Chromium and returns clean markdown or structured JSON, then call it over HTTP from your LLM app or agent. Firecrawl is the larger stack (eight services, an API-key gateway, separate workers). Crawl4AI is the smaller one (a crawler service plus Redis, with an MCP endpoint). Both run on Railway from a template, and both need a paid plan.

Deploy Firecrawl on Railway

When to run your own crawler

Language models need page text without the navigation, scripts and ads. A crawling API does that conversion for you. Typical uses:

Running it yourself keeps requests on infrastructure you control. It also means you carry the legal and operational responsibility listed below.

Scraping templates people deploy on Railway

TemplateCreatorDeploymentsServices
Crawl4AIbeau's Projects2981
Firecrawl v2 Web Data API for AITracklayer48
Crawl4AI LLM-Ready Web CrawlerTracklayer22

Railway marketplace data as of 2026-11-19. Full ranking: Best web scraping and crawling templates on Railway.

Check each template’s services and pinned version. A self-hosted crawler with no authentication is open to anyone who finds the URL, so look for a generated API key or token.

Deploy Firecrawl, step by step

These steps use Tracklayer’s Firecrawl template (image ghcr.io/firecrawl/firecrawl:2.11.504).

  1. Deploy the template. It creates a gateway (the only public service), the API, an orchestration worker, scrape executors, a Playwright/Chromium service, a Postgres job queue, Redis and RabbitMQ. All databases stay on the private network.
  2. Find your key. Open the firecrawl gateway service, then Variables, and copy FIRECRAWL_API_KEY.
  3. Make a first request. Use the gateway’s public URL: curl -X POST https://<gateway-domain>/v2/scrape -H "Authorization: Bearer <key>" -H "Content-Type: application/json" -d '{"url":"https://example.com","formats":["markdown"]}' In an SDK, pass the same URL as api_url.
  4. Optional: JSON extraction. Set OPENAI_API_KEY on firecrawl-api, and optionally OPENAI_BASE_URL and MODEL_NAME, to enable structured extraction and /extract. Plain scraping needs no LLM key.

Deploy Crawl4AI, step by step

These steps use Tracklayer’s Crawl4AI template (based on unclecode/crawl4ai:0.9.4).

  1. Deploy the template. It creates the Crawl4AI service and a Redis service with a volume for job results.
  2. Find your token. Open the Crawl4AI service, then Variables, and copy CRAWL4AI_API_TOKEN.
  3. Make a first request: curl -X POST https://<your-domain>/md -H "Authorization: Bearer <token>" -H "Content-Type: application/json" -d '{"url": "https://example.com"}'
  4. Use the extras. /playground builds requests in the browser and /dashboard shows monitoring. For MCP clients, the endpoint is /mcp/sse.
  5. Run long crawls as jobs. Railway ends single HTTP requests after 15 minutes, so use POST /crawl/job and poll GET /crawl/job/{task_id}.

LLM-based extraction in Crawl4AI needs an OpenAI, Anthropic or Gemini key. Plain crawling needs none.

Persistence

Firecrawl keeps its job queue in Postgres and its queues and crawl state in Redis, both on volumes. Crawl4AI keeps job status and results in a Redis service, so they survive redeploys. Crawl4AI results expire after REDIS_TASK_TTL seconds.

Costs

Both templates are memory-heavy because of headless Chromium. The figures below are the templates’ own estimates from Railway’s list prices, not invoices.

Idle estimate Typical estimate Minimum plan
Firecrawl about $30 per month (about 2.6 GB RAM) about $121 per month (about 6.9 GB RAM, a few thousand pages per day) Hobby
Crawl4AI about $7 per month (about 0.6 GB RAM) about $31 per month (about 2 GB RAM) Hobby

Firecrawl load follows crawl volume, and outbound fetches count as Railway egress. For Crawl4AI, raise CRAWL4AI_MAX_PAGES only as memory allows.

Scaling and upgrades

Licences and responsible use

Other options

Compare more crawlers in the full ranking of web scraping templates on Railway. If you want to send the results into workflows, see n8n templates.

FAQ

How do I authenticate to a self-hosted Firecrawl?

Copy FIRECRAWL_API_KEY from the firecrawl gateway service's variables and send it as an Authorization Bearer header to the gateway's public URL. The gateway exists because a self-hosted Firecrawl without its cloud authentication backend accepts requests from anyone who knows the URL.

How much does Firecrawl cost to run on Railway?

The template's own estimates, based on Railway's list prices, are about $30 per month idle and about $121 per month under regular crawling of a few thousand pages per day. It runs eight services, and load follows crawl volume. Outbound fetches count as Railway egress. The README says Hobby is the minimum plan because Trial's 1 GB per service is too small.

How much does Crawl4AI cost to run on Railway?

The template's own estimates are about $7 per month idle and about $31 per month under typical use, rising with CRAWL4AI_MAX_PAGES. Memory from the browsers drives cost: about 0.9 GB was measured locally after a few crawls with two pooled browsers, and each concurrent page adds roughly 50 to 150 MB. Hobby is the minimum plan.

What are the licences?

Firecrawl is AGPL-3.0; the template runs its images unmodified, and AGPL obligations apply if you modify Firecrawl and offer it over a network. Crawl4AI is Apache-2.0 with an additional attribution requirement for UncleCode. Both templates are community templates, not affiliated with the upstream projects. Ask a lawyer if you are unsure.

Am I allowed to scrape any website?

No. You are responsible for following the terms of service, robots.txt rules and applicable law of every site you crawl. Railway's Fair Use Policy forbids scrapers that violate the terms of the sites they target. Only crawl content you are permitted to access.