Status page and incidents

From resource health to instance healthCopy Anchor Link

The worker's health monitor checks every active resource through its provider adapter at a fixed interval: HEALTH_CHECK_INTERVAL_SECONDS, 30 by default. Instance health then follows from the health of its components:

Services and databases are critical by default. Caches, queues and storage are not. A template can change this per resource with the critical field.

The status pageCopy Anchor Link

Each tenant has a status page, at /status/<tenantId> in the console. It groups resources into components that customers see: API, Workers, Database, Cache, Messaging, Storage, and DNS & TLS. The headline shows the worst component state.

Who can see itCopy Anchor Link

Tenant owners and admins choose under Settings on the status page (/status/<tenantId>/settings):

A public page shows the title and description you set, component names and states, incidents and maintenance. It never shows resource ids, providers, regions or alert details. Incidents that OrkStage opened on its own read as a plain impact, such as "Degraded performance on Acme production", and their automatic updates as plain sentences. Incidents and updates written by people appear as written. Resolved incidents stay listed for 14 days. The public address is answered with its own rate limits, and an address that is not public looks exactly like one that does not exist.

ComponentsCopy Anchor Link

The same settings page renames components (for example API to Checkout), hides them or changes their order. Their status still follows the resources behind them.

Email updatesCopy Anchor Link

Anyone who can read the page can get an email when an incident opens, changes or resolves, and when maintenance is scheduled, starts and ends:

• In the console, Email me updates on the status page subscribes your own address at once.
• On a public page, people enter an address and confirm it from the email they receive.

Every email ends with an unsubscribe link. People who subscribed on the public page stop getting emails while the page is not public. The settings page shows how many confirmed subscribers there are. Emails go through the SMTP server of the platform (see configuration).

Automated incidentsCopy Anchor Link

Alert rules decide when OrkStage opens an incident on its own. There are two kinds:

• Health rules count failed health checks in a row.
• Metric rules compare a statistic of a metric (average, maximum or minimum) over a window of minutes with a threshold, for example "average CPU % above 80 over 5 minutes".

OrkStage seeds two health rules:

Incidents follow three rules:

• They are deduplicated per rule and resource. A resource has at most one open incident for a rule, so a flapping resource never opens many.
• After an incident resolves, a cooldown passes before a new one can open for the same resource and rule.
• They are resolved automatically when health returns or the metric is back within the threshold, with a final update. A rule with Resolve when it clears turned off leaves the incident open for a person to close.

Managing alert rulesCopy Anchor Link

Alert rules, in the console menu (/settings/alerts), lists the rules that apply to you and lets you add, edit, turn off and delete them.

• Platform staff manage every rule. A rule without a tenant is a platform rule and applies to every tenant.
• Tenant owners and admins manage their own tenant's rules. Other members read them.

A metric rule has these fields:

The worker reads each metric rule at most once a minute per instance, because provider metric APIs are billed per request. When the provider has no data for the window, the rule waits.

Where the metrics come from:

The same rules are available through the API at /alert-rules (GET, POST, PATCH, DELETE). Reading needs the instances:read scope and changing needs incidents:write.

{
  "tenantId": "<tenant id>",
  "name": "API CPU above 80%",
  "condition": {
    "signal": "metric",
    "metric": "cpu",
    "resourceTypes": ["compute.container-service"],
    "statistic": "avg",
    "operator": ">",
    "threshold": 80,
    "windowMinutes": 5
  },
  "severity": "high",
  "cooldownSeconds": 900
}

Trying a metric alert on a simulated targetCopy Anchor Link

Pin a metric of a resource to a value, wait for the rule to fire, then let it go back:

# CPU of the resource "web" stays at 95 until you clear it
curl -X POST "$ORK_API_URL/api/v1/instances/$INSTANCE/simulate" \
  -H "Authorization: Bearer $ORK_TOKEN" -H 'Content-Type: application/json' \
  -d '{"resourceId":"web","metric":{"name":"cpu","value":95}}'

# back to the generated series
curl -X POST "$ORK_API_URL/api/v1/instances/$INSTANCE/simulate" \
  -H "Authorization: Bearer $ORK_TOKEN" -H 'Content-Type: application/json' \
  -d '{"resourceId":"web","metric":{"name":"cpu","value":null}}'

The incident opens within a minute or two of the pin, depending on the window, and resolves the same way after you clear it.

Manual incidents and updatesCopy Anchor Link

Platform staff (owners, admins and operators) can open an incident by hand and post updates. Each update has a status, a message and an optional impact. Tenant members can read incidents and their updates, which appear on the incident page in order. Through the API, this needs the incidents:write scope.

Maintenance windowsCopy Anchor Link

Owners, admins and platform staff schedule maintenance under Settings on the status page: a title, a message, when it starts and ends, and which instance and components it covers (all of them when none are chosen).

• Before it starts, the status page lists it as scheduled.
• While it runs, its components show MAINTENANCE, and their resources open no incidents from health or metric rules.
• Subscribers and the tenant's owners and admins are told when it is scheduled, when it starts and when it ends.
• Cancelling a window that has not started deletes it. Ending a running one stops it now and keeps it in the history.

Through the API: GET, POST and PATCH /maintenance-windows, and DELETE /maintenance-windows/:id to cancel or end one. Changing them needs the incidents:write scope.