Observability
Follow an OrkStage installation through JSON logs with correlation ids, health endpoints, deployment events and the audit log. Metrics and tracing are planned.
Logs
The API and the worker write structured JSON logs (Pino) to stdout. Each line carries service and
context fields and the correlation id of the request. Tokens, secrets, cookies and
authorization headers are redacted. LOG_LEVEL sets the level (info by default).
docker compose --profile platform logs -f api workerThe correlation id starts at the HTTP request (x-correlation-id) and travels into the deployment,
the OpenTofu runs, the audit events and the webhook deliveries. Search your log store for it to
follow one operation end to end.
Health endpoints
| Endpoint | Meaning |
|---|---|
GET /api/v1/healthz | Liveness: the API process answers. |
GET /api/v1/readyz | Readiness: Postgres and Redis are reachable. The Compose health check calls it. |
Operational data in the product
- Deployments: every step with its duration, attempts and failure class, plus the event log and the Gantt timeline.
- Overview: instances, deployments, failures and incidents, with trends.
- Audit log: every change, filtered by tenant (
GET /audit). - Webhook deliveries: status, attempts and the last error of each endpoint.
Metrics and tracing
The API and the worker export OpenTelemetry traces and metrics over OTLP when
OTEL_EXPORTER_OTLP_ENDPOINT is set (for example http://otel-collector:4318). Without it, nothing
is loaded and nothing is exported. OTEL_EXPORTER_OTLP_HEADERS and the other standard OTEL_*
variables work as usual, so a hosted backend (Grafana Cloud, Honeycomb, Datadog) can receive the
data directly.
Traces
- Incoming HTTP requests (the health probes are left out), outgoing
fetchcalls (provider APIs, webhooks), Redis and Prisma queries. - Queue jobs continue the trace of the request that queued them: a deployment shows as one trace
from
POST /instances/:id/deploythrough the worker'sjob deploymentsspan. - Spans carry
tenant.id,instance.id,deployment.idanddeployment.kindwhere they apply.
Metrics
| Metric | Unit | Attributes |
|---|---|---|
http.server.request.duration (request count and duration) | s | method, status code |
orkstage.queue.depth | jobs | queue, state |
orkstage.job.duration | s | queue, outcome |
orkstage.job.retries | count | queue |
orkstage.deployment.duration (success and failure rates from its outcome) | s | kind, outcome |
orkstage.provider.request.duration | s | provider, operation, outcome |
orkstage.provider.errors | count | provider, operation, failure.class |
orkstage.health_check.count (failure rate: share not HEALTHY) | count | provider, state |
orkstage.incident.opened | count | source |
orkstage.incident.time_to_resolve (MTTR) | s | |
orkstage.notification.failures | count | channel |
Try it locally
pnpm platform:observe # the stack plus an OpenTelemetry Collector and JaegerDeploy an instance, then open Jaeger at http://localhost:16686 (service orkstage-api) for the trace,
and http://localhost:8889/metrics for the metrics in Prometheus format. The collector configuration
is otel-collector.yaml in the development workspace.
Service level objectives
Targets for running OrkStage, not guarantees to customers. Each one names the metric that measures it.
| Objective | Target | Measured with |
|---|---|---|
| API availability | 99.9% of requests without a 5xx | http.server.request.duration by status |
| Control plane read latency | p95 under 300 ms for GET | http.server.request.duration |
| Deployment orchestration | over 99% succeed, not counting provider side failures | orkstage.deployment.duration by outcome, with orkstage.provider.errors |
| Notification delivery | over 99% delivered | orkstage.notification.failures against notifications sent |
| Status page availability | at least the API's | http.server.request.duration for /status |
Alert on a burn rate (for example 2% of the monthly error budget in one hour), not on single
failures. Until dashboards exist, also alert on readyz, container restarts and deployment.failed
webhooks.
Backups and recovery
What to back up when you host OrkStage (Postgres, OpenTofu state, secrets), how to recover, and what OrkStage does for the backups of managed instances.
Troubleshooting
Fixes for common OrkStage problems, from startup configuration errors and busy ports to MiniStack, OpenTofu runners, state buckets, permissions and webhooks.