Crawl4AI LLM-Ready Web Crawler
Deploys the Crawl4AI 0.9 server, which turns web pages into Markdown and structured data for LLMs, RAG pipelines and agents. It runs headless Chromium behind a token-protected REST API with an MCP endpoint, a playground and a dashboard, plus Redis for background job results.
What you get
- Crawl4AI 0.9.4 (official image pinned by version and digest) with a pool of headless Chromium browsers and 1 GiB of shared memory
- REST API, MCP endpoints (/mcp/sse and /mcp/ws), a browser playground and a monitoring dashboard
- A generated CRAWL4AI_API_TOKEN and signing key
- Redis on a volume so async job status and results survive redeploys
- Variables for browser pool size, idle timeout, memory guard, job workers and rate limit
Good for
- Convert documentation sites and articles to Markdown for RAG ingestion
- Give Claude Code or other MCP clients a web-crawling tool
- Extract structured JSON from pages with an LLM key you provide
- Run crawls, screenshots and PDFs from n8n, Make or your backend over HTTP
Cost
Memory from the browsers drives cost. Measured locally: about 0.9 GB after a few crawls with two pooled browsers, and each concurrent page adds roughly 50 to 150 MB.
Services in this template
- Redis · image
redis:8.2 - Crawl4AI · from
baranberkay96/crawl4ai-railway
Questions people ask
How do I authenticate?
Copy CRAWL4AI_API_TOKEN from the service's Variables tab and send it as a Bearer token, for example to POST /md with a JSON body containing the URL. POST /token returns a 60-minute client token that can crawl but cannot run admin actions. The playground is at /playground and monitoring at /dashboard.
How do I run long crawls?
Railway ends single HTTP requests after 15 minutes. Use POST /crawl/job and poll GET /crawl/job/{task_id}; results stay in Redis for REDIS_TASK_TTL seconds.
Is an LLM key required?
Only for LLM-based extraction (an OpenAI, Anthropic or Gemini key). Plain crawling needs none.
How do I tune or scale it?
CRAWL4AI_MAX_PAGES caps concurrent browser pages, CRAWL4AI_BROWSER_IDLE_TTL returns memory after bursts and CRAWL4AI_JOB_WORKERS sets background concurrency. Scale vertically first; more replicas work for the REST API and polling, but keep one replica if you use MCP over SSE. To upgrade, change the image tag and digest in services/crawl4ai/Dockerfile.
Are there licence or usage terms?
Crawl4AI is Apache-2.0 with an additional attribution requirement for UncleCode. You are responsible for complying with the terms and robots rules of the sites you crawl, applicable law and Railway's Fair Use Policy. Internal and private network addresses are blocked by default.
Full template README
Deploy and Host Crawl4AI with Railway
Crawl4AI turns web pages into clean Markdown and structured data for LLMs, RAG pipelines and AI agents. This template runs the Crawl4AI 0.9 server with headless Chromium, a token-protected REST API, an MCP endpoint, the browser playground and monitoring dashboard, plus Redis for background job results.
About Hosting Crawl4AI
The Crawl4AI server is a FastAPI app that drives a pool of headless Chromium browsers. Hosting it well means enough memory and shared memory for the browsers, authentication from the first request, a stable signing key for tokens, and somewhere durable for async job results. This community template builds on the official unclecode/crawl4ai image pinned by version and digest, generates the API token and signing key, gives Chromium 1 GiB of /dev/shm, checks /health before routing traffic, and stores job status and results in a Railway Redis so they survive redeploys. Browser pool size, idle timeout, memory guard, job workers and rate limit are adjustable through variables.
Common Use Cases
- Converting documentation sites and articles to Markdown for RAG ingestion
- Giving Claude Code or other MCP clients a web-crawling tool through
/mcp/sse - Extracting structured JSON from pages with an LLM (OpenAI, Anthropic or Gemini key required)
- Running crawls from n8n, Make or your backend over plain HTTP with async jobs and polling
- Server-side screenshots and PDFs of web pages
Dependencies for Crawl4AI Hosting
- Redis for job results and monitor statistics (included)
- Optional: an LLM provider API key for LLM-based extraction; plain crawling needs none
Deployment Dependencies
- Crawl4AI repository: https://github.com/unclecode/crawl4ai
- Self-hosting guide: https://docs.crawl4ai.com/core/self-hosting/
- Docker deployment notes (0.9.4): https://github.com/unclecode/crawl4ai/blob/v0.9.4/deploy/docker/README.md
- Railway shared memory variable (
RAILWAY_SHM_SIZE_BYTES): https://docs.railway.com/reference/variables
Implementation Details
| Service | Source | Purpose |
|---|---|---|
| Crawl4AI | Dockerfile based on unclecode/crawl4ai:0.9.4 |
REST API, MCP, playground and dashboard on port 11235, healthcheck /health |
| Redis | redis:8.2 with a volume |
Job status and results (/crawl/job, /llm/job), dashboard statistics |
First request
- Copy
CRAWL4AI_API_TOKENfrom the Crawl4AI service's Variables tab and the domain from Settings → Networking. - Send it as a Bearer token:
curl -X POST https://{your-domain}/md -H "Authorization: Bearer {token}" -H "Content-Type: application/json" -d '{"url": "https://example.com"}'. - Open
https://{your-domain}/playgroundto build requests in the browser (paste the token there) and/dashboardfor live monitoring. - MCP:
claude mcp add --transport sse crawl4ai https://{your-domain}/mcp/sse --header "Authorization: Bearer {token}". A WebSocket transport is also available at/mcp/ws. - Client tokens:
POST /tokenwith{"email": "{you@company.com}", "api_token": "{token}"}returns a 60-minute token that can crawl but cannot run admin actions.
Long crawls: Railway ends single HTTP requests after 15 minutes. Use POST /crawl/job and poll GET /crawl/job/{task_id}; results stay in Redis for REDIS_TASK_TTL seconds.
Tuning and scaling: CRAWL4AI_MAX_PAGES caps concurrent browser pages (each needs roughly 50-150 MB), CRAWL4AI_BROWSER_IDLE_TTL returns memory after bursts, CRAWL4AI_JOB_WORKERS sets background concurrency. Scale vertically first. More replicas work for the REST API and job polling because Redis is shared; keep one replica if you rely on MCP over SSE.
Pinning and upgrades: the image tag and digest are in services/crawl4ai/Dockerfile. Change both, check the upstream migration notes, and redeploy.
Responsible use: you are responsible for complying with the terms of service and robots rules of the sites you crawl, applicable law, and Railway's Fair Use Policy. Internal and private network addresses are blocked by default.
This product includes software developed by UncleCode (https://x.com/unclecode) as part of the Crawl4AI project (https://github.com/unclecode/crawl4ai).
Why Deploy Crawl4AI on Railway?
Railway runs the browser-heavy crawler with HTTPS, private networking to Redis and your other services, and usage-based billing for the RAM and CPU it actually uses. Your agents and workflows get a crawling API you control, without managing servers.
Guides for this template
View Crawl4AI LLM-Ready Web Crawler on railway.com
Similar templates
| Template | Category | DeploymentsDeploys |
|---|---|---|
| 9Router Self-hosted AI router for Claude, Gemini, Cursor & OpenAI tools. | AI/ML | 9,734 |
| flowise-railway Flowise - low-code LLM apps builder | AI/ML | 8,512 |
| LibreChat Open-source ChatGPT clone: multi-model, agents, search, multi-user auth | AI/ML | 6,762 |
| Hermes Agent by Nous Research Template for Hermes Agent by Nous Research | AI/ML | 6,413 |
| OpenClaw [Homebrew support] The AI that actually does things | AI/ML | 5,167 |
| Hermes Agent Self Host Hermes Agent. Autonomous AI agent w/ self-improving, tools & more | AI/ML | 4,092 |