Crawl4AI on Railway

Open-source crawler that turns web pages into LLM-ready Markdown, with a token-protected API on Railway.

Official site · License: Apache-2.0 with attribution requirement · 7 Railway templates

Deploy Crawl4AI on RailwayCompare templates

Crawl4AI is an open-source web crawler and scraper built for LLM pipelines. It drives headless Chromium, renders JavaScript-heavy pages and returns clean Markdown or structured data that you can feed into RAG systems, agents and data pipelines.

The server mode exposes a REST API, an MCP endpoint for tools such as Claude Code, a browser playground and a monitoring dashboard. It is often compared with hosted crawling services such as Firecrawl, with the difference that you run it yourself and keep the crawled data on your own infrastructure.

Crawl4AI is released under Apache-2.0 with an additional attribution requirement.

What Crawl4AI can do

  • Headless Chromium crawling with JavaScript rendering
  • Markdown output suited to LLMs and RAG
  • REST API with async crawl jobs
  • MCP endpoints at /mcp/sse and /mcp/ws
  • LLM-based structured extraction with your own provider key
  • Screenshots and PDFs of pages
  • Playground and monitoring dashboard

What people use it for

  • Converting documentation sites to Markdown for RAG ingestion
  • Giving AI agents a web-crawling tool through MCP
  • Extracting structured JSON from pages
  • Running crawls from n8n, Make or your own backend over HTTP

Videos

Crawl4AI Official Tutorial, Full 1hr with Quickstart Examples · Unclecode
Turn ANY Website into LLM Knowledge in SECONDS · Cole Medin
Crawl4AI: The Ultimate AI Website Scraping Guide · Mervin Praison

Running Crawl4AI on Railway

On Railway, the Crawl4AI template deploys the official image, pinned by version and digest, as one service with 1 GiB of shared memory for Chromium, plus a Redis service that stores async job status and results. The API token and signing key are generated at deploy time. Browser memory drives cost; the template author measured about 0.9 GB after a few crawls with two pooled browsers, and each concurrent page adds roughly 50 to 150 MB.

Crawl4AI templates on Railway

Crawl4AI LLM-Ready Web CrawlerMaintained by Tracklayer
Deploys the Crawl4AI 0.9 server, which turns web pages into Markdown and structured data for LLMs, RAG pipelines and agents. It runs headless Chromium behind a token-protected REST
2 deployments
Deploy
Crawl4AI by beau's Projects
Crawl4AI is an AI-ready web crawler for LLMs, AI agents, & data pipelines.
298 deployments
Deploy
Crawl4AI by an independent creator
Open-source LLM-friendly web crawler with REST API and MCP integration.
13 deployments
Deploy
Crawl4AI by an independent creator
Web crawler API that turns pages into LLM-ready Markdown, with MCP
2 deployments
Deploy
Crawl4AI by an independent creator
Open-source web crawler API that turns pages into LLM-ready markdown
1 deployments
Deploy
Crawl4AI by an independent creator
Open-source LLM-friendly web crawler with REST API, playground and MCP.
1 deployments
Deploy
Crawl4AI | (Just Updated) LLM-Ready Web Crawler API, Token-Locked From Boot by an independent creator
Crawl4AI. Token-locked from boot, pinned image, health-checked, no fields
0 deployments
Deploy

Tracklayer templates are listed first. Other Crawl4AI templates follow, ordered by total deployments on Railway. How this works.

Deployment numbers as of 2026-11-20.

Questions about Crawl4AI

How do I authenticate against the API?

Copy CRAWL4AI_API_TOKEN from the service Variables tab and send it as a Bearer token. POST /token returns a 60-minute client token that can crawl but cannot run admin actions.

How do I run long crawls?

Railway ends single HTTP requests after 15 minutes, so use POST /crawl/job and poll GET /crawl/job/{task_id}. Results stay in Redis for REDIS_TASK_TTL seconds.

Do I need an LLM API key?

Only for LLM-based extraction, which needs an OpenAI, Anthropic or Gemini key. Plain crawling needs none.

What drives the cost?

Browser memory. CRAWL4AI_MAX_PAGES caps concurrent pages and CRAWL4AI_BROWSER_IDLE_TTL returns memory after bursts. Scale vertically first.

Are there licence or usage terms to know?

Crawl4AI is Apache-2.0 with an additional attribution requirement. You are responsible for following the terms and robots rules of the sites you crawl, applicable law and Railway's Fair Use Policy.

Guides

Similar apps

Tracklayer maintains the templates marked as such; Crawl4AI is a trademark of its owners and this site is not affiliated with them.