Health checks, metrics, tracing, logs
The backend exposes four observability surfaces. /Healthcheck answers
readiness probes. /metrics answers a Prometheus scrape. OpenTelemetry
exports traces to a collector. Logs print to stdout as text or JSON. The
probes, the metrics, and the logs work with no setup. Tracing sends
nothing until you set one environment variable.
When to use this
Section titled “When to use this”Set up observability when:
- You wire orchestrator probes to the backend.
- You scrape metrics or alert on them.
- You ship traces to a collector.
- You set the log format and level for the deployment.
Before you start
Section titled “Before you start”- The probes and
/metricssit on the backend port, on the private network. The public edge does not route/metrics. - The backend listens on
3000. The edge (Caddy) publishes the TLS port instead —3443by default. Every127.0.0.1:3000address on this page is the backend port, reachable from inside the Docker network only. Run acurlagainst it from a shell in the stack, for exampledocker compose -f docker/compose.prod.yml exec backend curl -sf http://127.0.0.1:3000/Healthcheck. - Tracing reads three standard OTLP environment variables. They are the one deliberate exception to the config-file rule: the OpenTelemetry shard reads them itself.
1. Wire the probes
Section titled “1. Wire the probes”| Route | Use | Behaviour |
|---|---|---|
GET /Healthcheck |
Readiness probe | Checks every database pool, Redis, and the stream consumers. Answers 200 or 503. |
GET /Healthcheck/live |
Liveness probe | No dependency checks. Always 200 while the process is alive. |
Use /Healthcheck as the readiness probe only. Dependency health flaps
it, and a restart loop helps no one. Use /Healthcheck/live as the
liveness probe: it fails only when the process is gone.
The readiness body names the part that fails:
{ "status": "ok", "service": { "env": "production", "instance_id": "01JD8ZQ7M5T3R8XK2VBN4QW7TY" }, "database": { "status": "ok", "error": null, "pools": [ { "database": "main", "role": "rw", "status": "ok", "error": null } ] }, "redis": { "status": "ok", "error": null }, "events": { "status": "ok", "consumers": [ { "name": "hr", "consecutive_failures": 0, "restarts": 0, "last_success_at": "2026-11-13T07:52:32Z", "last_error": null } ] }, "event_loss": { "publish_flush_failures": 0, "publish_events_dropped": 0, "messages_dropped": 0 }, "event_lag": { "max_lag": 0 }}service.instance_id is one id per boot. Match it against the boot log
when you debug one instance of many.
2. Read the event-loss counters
Section titled “2. Read the event-loss counters”Event delivery is at-most-once by decision. The event_loss object
makes the silent part visible:
| Key | Counts |
|---|---|
publish_flush_failures |
Failed publisher flush passes. |
publish_events_dropped |
Events discarded in those failed flushes. |
messages_dropped |
Messages a consumer dropped after a handler error. |
The counters are in-process and reset on restart. They are visibility, not accounting: PostgreSQL holds the durable state.
event_lag.max_lag is the largest consumer-group lag the last lag-check
pass observed. The pass runs every 10 minutes by default. A rising value
warns before a slow consumer loses entries to the stream trim.
3. Scrape the metrics
Section titled “3. Scrape the metrics”GET /metrics serves the Prometheus text format 0.0.4. It needs no
auth. Counters are per process, so scrape every instance and let
Prometheus do the aggregation.
Counters raised by the code:
| Metric | Labels | Counts |
|---|---|---|
facet_events_published_total |
— | Events published to the bus. |
facet_messages_processed_total |
consumer |
Messages a consumer group processed. |
facet_jobs_enqueued_total |
kind |
Job rows inserted. |
facet_jobs_claimed_total |
kind |
Job rows claimed by a worker. |
facet_jobs_succeeded_total |
kind |
Job rows that finished succeeded. |
facet_jobs_failed_total |
kind |
Job rows that finished failed. |
facet_jobs_abandoned_total |
kind |
Rows the lease sweep recovered. |
facet_http_requests_total |
method, route, status_class |
HTTP requests, by final status class. |
facet_db_checkout_total |
database, role |
Connection-pool checkouts. The pool-sizing input. |
facet_cron_task_failures_total |
task |
Cron task runs that raised. |
facet_cron_task_lease_skips_total |
task |
Dispatches another instance already fired. |
facet_fiber_restarts_total |
fiber |
Supervised fiber restarts. |
facet_fiber_loop_errors_total |
fiber |
Long-lived loop turns that rescued an error. |
Gauges and counters read from live state at each scrape:
| Metric | Type | Source |
|---|---|---|
facet_event_loss_publish_flush_failures_total |
counter | The event_loss publisher counter. |
facet_event_loss_publish_events_dropped_total |
counter | The event_loss dropped-events counter. |
facet_event_loss_messages_dropped_total |
counter | The event_loss dropped-messages counter. |
facet_consumer_consecutive_failures{name} |
gauge | One stream consumer’s failure streak. |
facet_consumer_restarts_total{name} |
counter | One stream consumer’s supervised restarts. |
facet_event_lag_max |
gauge | The max consumer-group lag of the last lag-check pass. |
4. Turn on tracing
Section titled “4. Turn on tracing”Set three environment variables before the boot:
export OTEL_SERVICE_NAME=facetexport OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.internal:4318export OTEL_RESOURCE_ATTRIBUTES=environment=productionWith no endpoint set, the process builds spans and drops them. That is the safe default for dev and test.
Every HTTP request opens a root span named http.server.request. Every
SQL statement opens a db.query span, with the SQL text truncated to
2000 characters. A span that ends in an error carries error status, so
failed writes are findable in the trace. Telemetry failures never fail
the request they observe.
5. Set the log format and level
Section titled “5. Set the log format and level”The optional logs: section drives the whole process:
| Key | Values | Default |
|---|---|---|
format |
text, json |
text |
level |
trace, debug, info, notice, warn, error, fatal |
info |
An unknown value fails the boot. The LOG_LEVEL environment variable
does not affect the app process; the file is the one source.
In json format, one object prints per line:
| Field | Content |
|---|---|
ts |
Entry timestamp, RFC-3339 UTC. |
level |
Severity label, for example INFO. |
source |
Log source, for example facet.jobs.manager. |
message |
The entry message. |
exception |
{class, message}, when the entry carries one. |
context |
The fiber-local log context, when set. |
Example
Section titled “Example”A complete production setup: JSON logs at info, tracing on, and a probe
check after a deploy.
logs: format: json level: infoexport OTEL_SERVICE_NAME=facetexport OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.internal:4318export OTEL_RESOURCE_ATTRIBUTES=environment=production
curl -sf http://127.0.0.1:3000/HealthcheckTroubleshooting
Section titled “Troubleshooting”/Healthcheck answers 503 with status: degraded.
A stream consumer reports consecutive failures, or a database pool or
Redis is down. Read events.consumers, database.pools, and redis in
the body. The consumer state self-heals on the next clean turn.
/Healthcheck is green but events do not react.
The consumer group may be gone, for example after a Redis data loss. The loop logs a WARN that names the group and the stream. A backend restart re-creates the groups at boot.
/metrics answers 404 through the public hostname.
The edge does not route it. Scrape the backend port on the private network,
for example http://127.0.0.1:3000/metrics from inside the Docker network.
From the host, run the scrape in the backend container:
docker compose -f docker/compose.prod.yml exec backend curl -sf http://127.0.0.1:3000/metrics.
No traces reach the collector.
OTEL_EXPORTER_OTLP_ENDPOINT is unset or unreachable. Without it the
process builds spans and discards them. The exporter speaks the OTLP
HTTP protocol.
A level change does nothing.
LOG_LEVEL does not drive the app process. Set logs.level in the
config file and restart.
Related
Section titled “Related”- Event delivery — the at-most-once contract the loss counters make visible.
- Background jobs and cron — the job kinds behind the job metrics.
- Deploying with containers — the image healthcheck that curls
/Healthcheck. - Database migrations — the boot chain the readiness probe guards.
- Internals on GitHub: metrics.md, telemetry.md, logging.md, event.md.