Grafana 13 LGTM Observability Stack

Grafana's observability stack for logs, traces and metrics. This template deploys Grafana 13.2.3, Loki, Tempo, Prometheus and Grafana Alloy as a single OpenTelemetry endpoint, with logs and traces stored in Railway Buckets.

What you get

  • Grafana with Loki, Tempo and Prometheus datasources provisioned and linked
  • Alloy receiving OTLP over the private network, or publicly with a bearer token
  • Loki and Tempo data in two Railway Buckets, with volumes for write-ahead logs
  • Prometheus with remote write, OTLP and scraping of extra targets
  • Default retention: 31 days logs, 14 days traces, 15 days metrics

Good for

  • Collecting traces, logs and metrics from Railway apps through one OTLP endpoint
  • Jumping from a slow trace to the logs and metrics of that service
  • Dashboards and alerts on service metrics
  • Receiving telemetry from apps hosted outside Railway

Cost

The template notes list idle figures measured locally, not on Railway: about 0.6 GB RAM for the whole stack, mostly Grafana at about 0.33 GB. Typical memory and bucket storage grow with ingest and with the number of instrumented services.

Services in this template

  • tempo · from baranberkay96/grafana-lgtm-railway
  • alloy · from baranberkay96/grafana-lgtm-railway
  • loki · from baranberkay96/grafana-lgtm-railway
  • prometheus · from baranberkay96/grafana-lgtm-railway
  • grafana · from baranberkay96/grafana-lgtm-railway

Questions people ask

How do I log in?

No input is needed at deploy. Open the grafana public domain and log in as admin with GF_SECURITY_ADMIN_PASSWORD from the grafana service variables. The password applies on first start; change it later in Grafana.

How do my services send data?

Set OTEL_EXPORTER_OTLP_ENDPOINT to the alloy variable of the same name, OTEL_EXPORTER_OTLP_PROTOCOL to http/protobuf, and an OTEL_SERVICE_NAME. From outside Railway, use the Alloy public domain with the OTLP_PUBLIC_TOKEN bearer token; requests without it get 401.

Is data kept across redeploys?

Yes. Logs and traces are in Railway Buckets, and Grafana, Prometheus and the write-ahead logs use volumes.

Can I run multiple replicas?

No. It is a single-node stack that scales vertically, and Grafana uses SQLite so it stays at one replica. Alloy is stateless.

How do I upgrade a component?

Change the FROM tag in that service's Dockerfile and redeploy. Read the release notes first, since Loki and Tempo major versions can change configuration. Grafana, Loki and Tempo are AGPL-3.0.

Full template README

Deploy and Host Grafana LGTM with Railway

LGTM is Grafana's open-source observability stack: Loki for logs, Grafana for dashboards, Tempo for traces and Mimir for metrics, here replaced by single-node Prometheus. This community template deploys Grafana, Loki, Tempo and Prometheus plus Grafana Alloy as a single OpenTelemetry endpoint, with logs and traces stored in Railway Buckets and the datasources linked to each other.

About Hosting Grafana LGTM

A self-hosted LGTM stack needs more than four containers: Loki and Tempo need object storage and retention, Prometheus needs a way to receive pushed metrics, applications need one place to send OpenTelemetry data, and Grafana needs datasources that link a slow trace to its logs and metrics. This template wires that up on Railway. Alloy receives OTLP from your services over the private network, or from anywhere through a token-protected public endpoint, and routes traces to Tempo, logs to Loki and metrics to Prometheus. Loki and Tempo keep their data in Railway Buckets, Tempo generates request-rate and service-graph metrics from your traces, and Grafana opens with everything provisioned.

Common Use Cases

  • Collect traces, logs and metrics from apps on Railway with any OpenTelemetry SDK, using one endpoint.
  • Jump from a slow or failing trace to the logs and request metrics of that service, and back.
  • See service dependencies and error rates in a service graph built from traces.
  • Scrape /metrics endpoints of your Railway services and build dashboards and alerts on them.
  • Receive telemetry from apps hosted outside Railway through the token-protected OTLP endpoint.

Dependencies for Grafana LGTM Hosting

  • Grafana 13.2.3
  • Loki 3.7.8 (single binary) with a Railway Bucket
  • Tempo 3.1.0 (monolithic mode) with a Railway Bucket
  • Prometheus v3.15.0
  • Grafana Alloy v1.20.1 (OpenTelemetry collector)

Deployment Dependencies

Implementation Details

Service Role Public Storage
grafana Dashboards, Explore, alerting; datasources provisioned and linked Yes Volume (SQLite)
alloy OTLP endpoint: traces to Tempo, logs to Loki, metrics to Prometheus Yes, port 4320, bearer token –
loki Logs No Volume (WAL) + LokiBucket
tempo Traces, span metrics and service graphs No Volume (WAL) + TempoBucket
prometheus Metrics: remote write, OTLP, scraping No Volume (TSDB)

First login

  1. Deploy the template. No input is required.
  2. Open the grafana public domain and log in as admin with GF_SECURITY_ADMIN_PASSWORD from the grafana variables (applied on first start; change it later in Grafana).
  3. Open Explore and pick Prometheus, Loki or Tempo. The stack's own metrics are there from the start.

Sending data from services in this project

Add to your service:

  • OTEL_EXPORTER_OTLP_ENDPOINT = ${{alloy.OTEL_EXPORTER_OTLP_ENDPOINT}} (OTLP/HTTP on port 4318; gRPC is on port 4317)
  • OTEL_EXPORTER_OTLP_PROTOCOL = http/protobuf
  • OTEL_SERVICE_NAME = {your-service-name}

Sending data from outside Railway

  • OTEL_EXPORTER_OTLP_ENDPOINT = https://{alloy-public-domain} (OTLP/HTTP only)
  • OTEL_EXPORTER_OTLP_HEADERS = Authorization=Bearer%20{OTLP_PUBLIC_TOKEN} (most SDKs decode %20 to a space; if yours does not, use a literal space)

Requests without the token are rejected with 401. Rotate the token by changing OTLP_PUBLIC_TOKEN on alloy and redeploying.

Other ways in

  • Prometheus scraping: set EXTRA_SCRAPE_TARGETS on prometheus to a comma-separated list such as ${{api.RAILWAY_PRIVATE_DOMAIN}}:8080 (services exposing /metrics).
  • Prometheus remote write: ${{prometheus.PROMETHEUS_REMOTE_WRITE_URL}}.
  • Loki push API for log shippers (Promtail-compatible clients, Vector, Fluent Bit): ${{loki.LOKI_PUSH_URL}}.

Retention: Loki LOKI_RETENTION_PERIOD (31 days), Tempo TEMPO_RETENTION (14 days), Prometheus PROMETHEUS_RETENTION_TIME (15 days) and PROMETHEUS_RETENTION_SIZE (4 GB).

Scaling: this is a single-node stack. Scale vertically: give Loki and Tempo more memory as ingest grows, and Prometheus more memory and volume as series grow. Alloy is stateless; raise ALLOY_MEMORY_LIMIT with its memory. Grafana uses SQLite, so keep one replica.

Pinning and upgrades: each version is pinned in services/{service}/Dockerfile. To upgrade, change the FROM tag in that Dockerfile, read the release notes (Loki and Tempo major versions can change configuration), and redeploy the service. Data in the buckets and volumes is kept.

Resources: about 0.6 GB RAM idle (around $7/month). A setup with 5 to 15 instrumented services typically uses about 2.5 GB RAM and 30 GB of bucket storage (around $35/month). Hobby plan works; Pro once Prometheus needs more than 5 GB of volume.

Why Deploy Grafana LGTM on Railway?

Railway runs Grafana, Loki, Tempo, Prometheus and Alloy as one project with private networking to the services you monitor, volumes and S3 buckets for storage, generated secrets and usage-based billing. Your telemetry stays in your own project, and instrumenting a service is a matter of referencing one variable.

Guides for this template

View Grafana 13 LGTM Observability Stack on railway.com

Similar templates

TemplateCategoryDeploymentsDeploys
Uptime Kuma
A fancy self-hosted monitoring tool
Observability1,456
Ubuntu
Ubuntu in your network for debugging
Observability181
SigNoz APM for Railway AppsTracklayer
Logs, traces and metrics in one OpenTelemetry-native Datadog alternative.
Observability3