Crawl4AI on Railway
Open-source crawler that turns web pages into LLM-ready Markdown, with a token-protected API on Railway.
Crawl4AI is an open-source web crawler and scraper built for LLM pipelines. It drives headless Chromium, renders JavaScript-heavy pages and returns clean Markdown or structured data that you can feed into RAG systems, agents and data pipelines.
The server mode exposes a REST API, an MCP endpoint for tools such as Claude Code, a browser playground and a monitoring dashboard. It is often compared with hosted crawling services such as Firecrawl, with the difference that you run it yourself and keep the crawled data on your own infrastructure.
Crawl4AI is released under Apache-2.0 with an additional attribution requirement.
What Crawl4AI can do
- Headless Chromium crawling with JavaScript rendering
- Markdown output suited to LLMs and RAG
- REST API with async crawl jobs
- MCP endpoints at /mcp/sse and /mcp/ws
- LLM-based structured extraction with your own provider key
- Screenshots and PDFs of pages
- Playground and monitoring dashboard
What people use it for
- Converting documentation sites to Markdown for RAG ingestion
- Giving AI agents a web-crawling tool through MCP
- Extracting structured JSON from pages
- Running crawls from n8n, Make or your own backend over HTTP
Videos
Running Crawl4AI on Railway
On Railway, the Crawl4AI template deploys the official image, pinned by version and digest, as one service with 1 GiB of shared memory for Chromium, plus a Redis service that stores async job status and results. The API token and signing key are generated at deploy time. Browser memory drives cost; the template author measured about 0.9 GB after a few crawls with two pooled browsers, and each concurrent page adds roughly 50 to 150 MB.
Crawl4AI templates on Railway
Tracklayer templates are listed first. Other Crawl4AI templates follow, ordered by total deployments on Railway. How this works.
Deployment numbers as of 2026-11-20.
Questions about Crawl4AI
How do I authenticate against the API?
Copy CRAWL4AI_API_TOKEN from the service Variables tab and send it as a Bearer token. POST /token returns a 60-minute client token that can crawl but cannot run admin actions.
How do I run long crawls?
Railway ends single HTTP requests after 15 minutes, so use POST /crawl/job and poll GET /crawl/job/{task_id}. Results stay in Redis for REDIS_TASK_TTL seconds.
Do I need an LLM API key?
Only for LLM-based extraction, which needs an OpenAI, Anthropic or Gemini key. Plain crawling needs none.
What drives the cost?
Browser memory. CRAWL4AI_MAX_PAGES caps concurrent pages and CRAWL4AI_BROWSER_IDLE_TTL returns memory after bursts. Scale vertically first.
Are there licence or usage terms to know?
Crawl4AI is Apache-2.0 with an additional attribution requirement. You are responsible for following the terms and robots rules of the sites you crawl, applicable law and Railway's Fair Use Policy.
Guides
Similar apps
Tracklayer maintains the templates marked as such; Crawl4AI is a trademark of its owners and this site is not affiliated with them.
