Firecrawl v2 Web Data API for AI

Firecrawl is an open-source API that turns websites into markdown, HTML, screenshots or structured JSON for LLM applications. This template deploys the self-hosted stack with separate workers, a Chromium renderer, queue databases and an API-key gateway.

What you get

  • Caddy gateway that only forwards requests carrying your generated API key
  • Firecrawl API, orchestration worker and scrape executor services
  • Playwright service running headless Chromium
  • Queue Postgres, Redis and RabbitMQ on the private network
  • Optional OpenAI-compatible key for JSON extraction and /extract

Good for

  • Converting documentation sites you may crawl into markdown for RAG
  • Giving AI agents a scrape tool through the Firecrawl SDKs or MCP
  • Extracting structured JSON from product or listing pages
  • Monitoring your own sites for content changes

Services in this template

  • playwright-service · image ghcr.io/firecrawl/playwright-service:latest@sha256:717699bf8a61d94451926c810d19ca1d36278192907e307bf5229a558b08fcc6
  • Redis · image redis:8.2
  • nuq-postgres · image ghcr.io/firecrawl/nuq-postgres:latest@sha256:9b638af78d99873bc0ba2b57c9cbcd01df6ce96efeaa72d36cfbe0f76521e8fc
  • firecrawl-api · image ghcr.io/firecrawl/firecrawl:2.11.504
  • firecrawl · from baranberkay96/firecrawl-railway
  • firecrawl-worker · from baranberkay96/firecrawl-railway
  • RabbitMQ · image rabbitmq:4.3.6-alpine
  • firecrawl-nuq-worker · from baranberkay96/firecrawl-railway

Questions people ask

How do I authenticate?

Copy FIRECRAWL_API_KEY from the firecrawl gateway service variables and send it as an Authorization Bearer header to the gateway's public URL.

Why a gateway?

A self-hosted Firecrawl without its cloud auth backend accepts requests from anyone who knows the URL. The gateway rejects requests without your key.

How do I scale?

Raise NUQ_WORKER_COUNT or add replicas on firecrawl-nuq-worker, and add replicas to playwright-service for heavier JavaScript rendering. Keep firecrawl-worker at one replica.

How do I upgrade?

Change the image tag on firecrawl-api and in services/firecrawl-workers/Dockerfile together. Playwright and nuq-postgres are pinned by digest, so refresh those digests too.

Any licence or usage limits?

Firecrawl is AGPL-3.0. You are responsible for following each target site's terms and robots.txt. Railway's Fair Use Policy forbids scrapers that violate the terms of the sites they target.

Full template README

Deploy and Host Firecrawl with Railway

Firecrawl is an open-source API that turns websites into clean markdown, HTML, screenshots or structured JSON for LLM applications. This community template deploys the self-hosted Firecrawl stack on Railway with separate workers, a headless Chromium renderer, its queue databases and an API-key gateway in front.

About Hosting Firecrawl

Self-hosted Firecrawl is more than one API container. Scrape and crawl requests are queued, executed by worker processes that fetch pages directly or through a Playwright/Chromium service, converted to markdown, and tracked in Redis and a dedicated Postgres queue (NuQ) with RabbitMQ notifications. Without its cloud authentication backend, a self-hosted instance also accepts requests from anyone who knows the URL. This template runs the official Firecrawl images with the API, orchestration workers and scrape executors as separate Railway services, keeps all databases on the private network, and adds a small gateway that only forwards requests carrying your generated API key.

Common Use Cases

  • Convert documentation sites, help centers and blogs you are permitted to crawl into markdown for RAG pipelines.
  • Give AI agents a scrape and search tool through the Firecrawl SDKs, MCP server or n8n/LangChain integrations.
  • Extract structured JSON from product or listing pages with an OpenAI-compatible model.
  • Monitor your own or partner sites for content changes on a schedule.

Dependencies for Firecrawl Hosting

  • Firecrawl API and worker image ghcr.io/firecrawl/firecrawl:2.11.504
  • Firecrawl Playwright service (headless Chromium) and NuQ Postgres images (pinned by digest)
  • Redis (Railway Redis 8.2) and RabbitMQ 4.3
  • Caddy 2.11 as the API-key gateway
  • Optional: an OpenAI-compatible API key for JSON extraction and /extract

Deployment Dependencies

Implementation Details

Service Role Public Storage
firecrawl Caddy gateway, checks Authorization: Bearer {key} Yes (port 8080) –
firecrawl-api Firecrawl HTTP API No –
firecrawl-worker Crawl orchestration, extract, NuQ prefetch and reconciler No –
firecrawl-nuq-worker Scrape executors (5 processes by default) No –
playwright-service Headless Chromium rendering No –
nuq-postgres NuQ job queue (Postgres + pg_cron) No Volume
Redis BullMQ queues, crawl state, rate limits No Volume
RabbitMQ Job wake-up notifications No –

First request

  1. Copy FIRECRAWL_API_KEY from the firecrawl (gateway) service variables.
  2. Use the gateway's public URL as the API URL in an SDK (for example FirecrawlApp(api_key=..., api_url="https://{gateway-domain}")) or call it directly: curl -X POST https://{gateway-domain}/v2/scrape -H "Authorization: Bearer {key}" -H "Content-Type: application/json" -d '{"url":"https://example.com","formats":["markdown"]}'
  3. The BullMQ queue dashboard is at https://{gateway-domain}/admin/{BULL_AUTH_KEY}/queues (BULL_AUTH_KEY is on firecrawl-api).
  4. To enable JSON extraction and /extract, set OPENAI_API_KEY (and optionally OPENAI_BASE_URL, MODEL_NAME) on firecrawl-api; the workers reference those values.

Scaling: raise NUQ_WORKER_COUNT on firecrawl-nuq-worker or add replicas to it for more parallel scrapes, and add replicas to playwright-service for heavier JavaScript rendering. Keep firecrawl-worker at one replica.

Pinning and upgrades: the Firecrawl version is the image tag on firecrawl-api and in services/firecrawl-workers/Dockerfile; change both together. Playwright and nuq-postgres are pinned by digest because upstream only publishes latest; refresh the digests when you upgrade.

Responsible use: you are responsible for complying with the terms of service, robots.txt rules and applicable laws of every site you scrape. Railway's Fair Use Policy forbids scrapers that violate the terms of the sites they target; only crawl content you are permitted to access.

Why Deploy Firecrawl on Railway?

Railway runs the eight services as one project with private networking, generated credentials, volumes for the queue databases and usage-based billing, so idle time stays cheap and crawl bursts scale with replicas. The gateway gives you a single authenticated HTTPS endpoint without managing servers.

Guides for this template

View Firecrawl v2 Web Data API for AI on railway.com

Similar templates

TemplateCategoryDeploymentsDeploys
9Router
Self-hosted AI router for Claude, Gemini, Cursor & OpenAI tools.
AI/ML9,734
flowise-railway
Flowise - low-code LLM apps builder
AI/ML8,512
LibreChat
Open-source ChatGPT clone: multi-model, agents, search, multi-user auth
AI/ML6,762
Hermes Agent by Nous Research
Template for Hermes Agent by Nous Research
AI/ML6,413
OpenClaw
[Homebrew support] The AI that actually does things
AI/ML5,167
Hermes Agent
Self Host Hermes Agent. Autonomous AI agent w/ self-improving, tools & more
AI/ML4,092