Observability — Metrics, Logs, Dashboards, Tracing

Endpoints

These endpoints are mounted outside /api/v1 so infra probes can call them without a JWT.

Metrics exposed

Implementation: apps/cms/src/routes/metrics.ts + the withMetrics() middleware.

Backend: prom-client. Process default metrics are collected only when the runtime exposes a working Node process.cpuUsage() implementation; Workers/Wrangler stubs are skipped safely.

Request path labels are normalized before recording to avoid high-cardinality labels from UUIDs and numeric IDs.

Tracing / Apache SkyWalking POC

The Docker/Node runtime supports an optional tracing bootstrap. It is disabled by default and controlled by environment variables:

LUMIBASE_TRACING_ENABLED=false
LUMIBASE_TRACING_PROVIDER=skywalking
LUMIBASE_SERVICE_NAME=lumibase-cms
OTEL_EXPORTER_OTLP_ENDPOINT=http://skywalking-oap:11800
LUMIBASE_TRACING_SAMPLING_RATIO=1

Enable the SkyWalking stack in Docker by including the optional compose override:

docker compose \
  -f docker/docker-compose.yml \
  -f docker/docker-compose.skywalking.yml \
  up --build

Disable tracing by omitting docker/docker-compose.skywalking.yml or setting LUMIBASE_TRACING_ENABLED=false.

Notes:

• Tracing is Node/Docker-only for this POC. Cloudflare Workers should use Cloudflare-native observability until a separate Workers tracing design exists.
• The OpenTelemetry SDK bootstrap must run before the Hono app import so auto-instrumentations can patch supported Node modules early.
• Changing the tracing provider/endpoint should be treated as a process-restart operation; hot reload is intentionally out of scope for this POC.
• Request spans use normalized paths and never copy Authorization, cookies, raw query strings, or request bodies into attributes.

Structured logging

Every request logs JSON with fields:

{
  "timestamp": "2025-...",
  "level": "info",
  "requestId": "req_xyz",
  "method": "GET",
  "path": "/api/v1/items/posts",
  "status": 200,
  "duration": 45,
  "siteId": "s_abc",
  "userId": "u_123"
}

• Cloudflare: logs are shipped via Workers Logpush to R2/S3/external.
• Docker: logs go to stdout, collected by Promtail/Loki or the runtime log driver.

Stack monitoring

Cloudflare mode

Use the built-in services:

• Workers Analytics Engine — emit custom metrics, query via the dashboard.
• Workers Logpush — log shipping.
• Cloudflare Trace for APM.

Documentation only — nothing is self-hosted.

Docker mode

The metrics/log dashboard stack runs via:

docker compose -f docker/docker-compose.yml -f docker/docker-compose.monitoring.yml up

The Tracing POC stack additionally runs the file docker/docker-compose.skywalking.yml: Config: docker/prometheus/prometheus.yml, docker/grafana/provisioning/, docker/grafana/dashboards/lumibase.json, docker/grafana/dashboards/agent-harness.json.

Pre-provisioned Grafana dashboard

The Lumibase dashboard (auto-loaded) includes:

• Request rate (req/s) over time, broken down by status class.
• Latency percentiles p50, p95, p99.
• Error rate (%) and top error codes.
• Queue depth for each queue (search-index, media-thumbnails, webhook).
• Cache hit ratio (%).
• DB pool utilization (active vs idle).
• CPU / Memory of the CMS container.

The Lumibase Agent Harness dashboard includes run success/fail rate, budget stop reason, tool latency, approval latency, evaluation outcome, token/cost estimate, and dead-letter enqueue rate. Repeatedly failing runs are pushed to the agent-dead-letter queue when the runtime queue adapter is available.

Backup monitoring

The pg-backup service runs pg_dump daily at 02:00 UTC, uploads to MinIO/S3 with a retention of 7 daily + 4 weekly. Failures send a notification via webhook (configurable).

Config: docker/prometheus/prometheus.yml, docker/grafana/provisioning/, docker/grafana/dashboards/lumibase.json.

Health check details

GET /health response:

{
  "status": "healthy",
  "services": {
    "database": "healthy",
    "cache": "healthy",
    "search": "healthy",
    "storage": "healthy",
    "queue": "healthy"
  }
}

GET /health always returns 200 so it is safe as a liveness/diagnostic endpoint. GET /health/ready returns 503 when any dependency probe is unhealthy, so orchestrators can use it for readiness.

Probe timeout: 750ms per dependency.

Alerting

Recommended alert rules in Prometheus:

• Error rate > 5% over 5 minutes.
• p95 latency > 500ms over 10 minutes.
• Health readiness failing.
• Queue job failures rising continuously.
• DB query duration p95 above the service SLO.

See apps/docs/content/guides/tooling-recommendations.md for managed alternatives (Datadog, Sentry, New Relic).