Web scraping API for LLM apps: Firecrawl and Crawl4AI
Short answer: self-host a crawling API that fetches pages in headless Chromium and returns clean markdown or structured JSON, then call it over HTTP from your LLM app or agent. Firecrawl is the larger stack (eight services, an API-key gateway, separate workers). Crawl4AI is the smaller one (a crawler service plus Redis, with an MCP endpoint). Both run on Railway from a template, and both need a paid plan.
Deploy Firecrawl on RailwayWhen to run your own crawler
Language models need page text without the navigation, scripts and ads. A crawling API does that conversion for you. Typical uses:
- turning documentation sites, help centres and blogs you may crawl into markdown for RAG,
- giving an AI agent a scrape and search tool through an SDK, MCP or n8n,
- extracting structured JSON from product or listing pages with an LLM,
- server-side screenshots and PDFs (Crawl4AI).
Running it yourself keeps requests on infrastructure you control. It also means you carry the legal and operational responsibility listed below.
Scraping templates people deploy on Railway
| Template | Creator | Deployments | Services |
|---|---|---|---|
| Crawl4AI | beau's Projects | 298 | 1 |
| Firecrawl v2 Web Data API for AI | Tracklayer | 4 | 8 |
| Crawl4AI LLM-Ready Web Crawler | Tracklayer | 2 | 2 |
Railway marketplace data as of 2026-11-19. Full ranking: Best web scraping and crawling templates on Railway.
Check each template’s services and pinned version. A self-hosted crawler with no authentication is open to anyone who finds the URL, so look for a generated API key or token.
Deploy Firecrawl, step by step
These steps use Tracklayer’s Firecrawl template (image ghcr.io/firecrawl/firecrawl:2.11.504).
- Deploy the template. It creates a gateway (the only public service), the API, an orchestration worker, scrape executors, a Playwright/Chromium service, a Postgres job queue, Redis and RabbitMQ. All databases stay on the private network.
- Find your key. Open the
firecrawlgateway service, then Variables, and copyFIRECRAWL_API_KEY. - Make a first request. Use the gateway’s public URL:
curl -X POST https://<gateway-domain>/v2/scrape -H "Authorization: Bearer <key>" -H "Content-Type: application/json" -d '{"url":"https://example.com","formats":["markdown"]}'In an SDK, pass the same URL asapi_url. - Optional: JSON extraction. Set
OPENAI_API_KEYonfirecrawl-api, and optionallyOPENAI_BASE_URLandMODEL_NAME, to enable structured extraction and/extract. Plain scraping needs no LLM key.
Deploy Crawl4AI, step by step
These steps use Tracklayer’s Crawl4AI template (based on unclecode/crawl4ai:0.9.4).
- Deploy the template. It creates the Crawl4AI service and a Redis service with a volume for job results.
- Find your token. Open the Crawl4AI service, then Variables, and copy
CRAWL4AI_API_TOKEN. - Make a first request:
curl -X POST https://<your-domain>/md -H "Authorization: Bearer <token>" -H "Content-Type: application/json" -d '{"url": "https://example.com"}' - Use the extras.
/playgroundbuilds requests in the browser and/dashboardshows monitoring. For MCP clients, the endpoint is/mcp/sse. - Run long crawls as jobs. Railway ends single HTTP requests after 15 minutes, so use
POST /crawl/joband pollGET /crawl/job/{task_id}.
LLM-based extraction in Crawl4AI needs an OpenAI, Anthropic or Gemini key. Plain crawling needs none.
Persistence
Firecrawl keeps its job queue in Postgres and its queues and crawl state in Redis, both on volumes. Crawl4AI keeps job status and results in a Redis service, so they survive redeploys. Crawl4AI results expire after REDIS_TASK_TTL seconds.
Costs
Both templates are memory-heavy because of headless Chromium. The figures below are the templates’ own estimates from Railway’s list prices, not invoices.
| Idle estimate | Typical estimate | Minimum plan | |
|---|---|---|---|
| Firecrawl | about $30 per month (about 2.6 GB RAM) | about $121 per month (about 6.9 GB RAM, a few thousand pages per day) | Hobby |
| Crawl4AI | about $7 per month (about 0.6 GB RAM) | about $31 per month (about 2 GB RAM) | Hobby |
Firecrawl load follows crawl volume, and outbound fetches count as Railway egress. For Crawl4AI, raise CRAWL4AI_MAX_PAGES only as memory allows.
Scaling and upgrades
- Firecrawl: raise
NUQ_WORKER_COUNTor add replicas tofirecrawl-nuq-worker, and add replicas toplaywright-servicefor heavier JavaScript pages. Keepfirecrawl-workerat one replica. To upgrade, change the image tag onfirecrawl-apiand inservices/firecrawl-workers/Dockerfiletogether, and refresh the pinned Playwright and Postgres image digests. - Crawl4AI: scale vertically first. More replicas work for the REST API and job polling, but keep one replica if you use MCP over SSE. To upgrade, change the image tag and digest in
services/crawl4ai/Dockerfile.
Licences and responsible use
- Firecrawl is AGPL-3.0. The template uses the images unmodified; AGPL obligations apply if you modify Firecrawl and offer it over a network.
- Crawl4AI is Apache-2.0 with an additional attribution requirement for UncleCode. Crawl4AI blocks internal and private network addresses by default.
- Both are community templates, not affiliated with the upstream projects.
- You must follow each site’s terms, robots.txt and applicable law. Railway’s Fair Use Policy forbids scrapers that violate those terms.
Other options
Compare more crawlers in the full ranking of web scraping templates on Railway. If you want to send the results into workflows, see n8n templates.
FAQ
How do I authenticate to a self-hosted Firecrawl?
Copy FIRECRAWL_API_KEY from the firecrawl gateway service's variables and send it as an Authorization Bearer header to the gateway's public URL. The gateway exists because a self-hosted Firecrawl without its cloud authentication backend accepts requests from anyone who knows the URL.
How much does Firecrawl cost to run on Railway?
The template's own estimates, based on Railway's list prices, are about $30 per month idle and about $121 per month under regular crawling of a few thousand pages per day. It runs eight services, and load follows crawl volume. Outbound fetches count as Railway egress. The README says Hobby is the minimum plan because Trial's 1 GB per service is too small.
How much does Crawl4AI cost to run on Railway?
The template's own estimates are about $7 per month idle and about $31 per month under typical use, rising with CRAWL4AI_MAX_PAGES. Memory from the browsers drives cost: about 0.9 GB was measured locally after a few crawls with two pooled browsers, and each concurrent page adds roughly 50 to 150 MB. Hobby is the minimum plan.
What are the licences?
Firecrawl is AGPL-3.0; the template runs its images unmodified, and AGPL obligations apply if you modify Firecrawl and offer it over a network. Crawl4AI is Apache-2.0 with an additional attribution requirement for UncleCode. Both templates are community templates, not affiliated with the upstream projects. Ask a lawyer if you are unsure.
Am I allowed to scrape any website?
No. You are responsible for following the terms of service, robots.txt rules and applicable law of every site you crawl. Railway's Fair Use Policy forbids scrapers that violate the terms of the sites they target. Only crawl content you are permitted to access.