Skip to content

Open uptime incidents only after a run of failed checks - #5

Merged
sinnercode228 merged 3 commits into
mainfrom
feature/uptime-failure-threshold
Oct 16, 2026
Merged

sinnercode228 merged 3 commits into
mainfrom
feature/uptime-failure-threshold

Conversation

@sinnercode228

Copy link
Copy Markdown
Owner

What

  • Monitors get a failureThreshold: failed checks in a row that open an incident. Default 2, the admin API accepts 1 to 10, and 1 keeps the old behaviour. The first successful check still closes the incident.
  • The incident starts at the first failed check of the run, and its failures count covers the whole run.
  • Every check is still written to checks and uptime_daily, so a single failure still lowers the uptime percentages.
  • Migration 2 adds monitors.failure_threshold INTEGER NOT NULL DEFAULT 2; existing databases apply it on start.
  • Migrations run under BEGIN IMMEDIATE and re-read user_version inside it, and busy_timeout is set first. Before, when the API and the uptime worker opened an existing database at once, one crashed with duplicate column name: failure_threshold.
  • POST /api/admin/monitors takes failureThreshold; GET /api/monitors returns it.
  • Before each check, the failed checks that have not opened an incident yet are read back from checks, so a restart or the separate worker keeps counting.
  • The demo runs the same core code with a threshold of 2.
  • README: limitation 5 reworded (no alerts, recovery on one success), API docs with a curl example, line links, test counts (118 to 129).

Why

One failed check opened an incident, so a single timeout showed up as an outage of its own.

How it is tested

  • Core: one failure opens nothing by default; two in a row open one that starts at the first; the cause comes from the first failure; a success resets the count; threshold 1 gives the old incidents; a run passed in as state continues.
  • Simulator: for all six demo monitors over 90 days, the incidents are exactly the runs of 2 or more failed checks. The demo engine test checks the threshold and failure counts.
  • Server: a timeout then a success opens nothing, then three 500s open one incident from the first; restarts on a file database mid-run keep counting; the admin API stores the threshold and rejects 0, 11 and 1.5; a version 1 database migrates on start, also when two processes open it together.
  • 13 new or changed tests failed on main. The two-process test failed 5 of 5 runs before the migration fix and passed 20 of 20 after it.
  • From a clean checkout: npm ci, lint, format check, typecheck, 129 tests and build pass on Node 22.22.2 and 24.22.0.

docs/screenshots/uptime.png still shows single-check incidents; retaking it here would change its fonts.

@sinnercode228
sinnercode228 merged commit 748f616 into main Oct 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant