Compare commits

..

24 Commits

Author SHA1 Message Date
Niklas Ye c5be55dcbc Set the chart's placeholder version to 0.28.1
CI / chart (push) Successful in 2s
CI / security (push) Successful in 15s
CI / test (push) Successful in 2m51s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 31s
Release / image (push) Successful in 59s
Release / scan-image (push) Successful in 3s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as 36c00ac and 9bf4c92, so a
tree heading for v0.28.1 doesn't say 0.28.0.
2026-09-26 10:21:54 +02:00
Niklas Ye b2c3868619 Fix the web UI stuck on its loading spinner
format.js defined isoWeek twice. 9d1df2b added a second copy for the
week numbers on the rota without noticing the first, which already
exported the same function. A duplicate function declaration is legal in
a plain script but an early SyntaxError in an ES module, so the browser
refused format.js, every module importing it, and with them app.js. Its
boot() is what hides the spinner, so nothing ever did.

Keeps the first definition, whose comment explains the Thursday rule, and
drops the second. Both give ISO 8601 numbers; the one kept was checked
against 2026-01-01 (week 1), 2026-09-28 (40), 2026-12-31 (53) and
2024-12-30 (1). oncall.js and team.js import the name unchanged.

The suite could not see this: it has no JS, and node --check reads a .js
file as a script, where the redeclaration passes. Checking each file as a
module (.mjs) does catch it.
2026-09-26 10:21:54 +02:00
Niklas Ye 36c00acf62 Set the chart's placeholder version to 0.28.0
CI / chart (push) Successful in 1s
CI / security (push) Successful in 17s
CI / test (push) Successful in 3m1s
Release / test (push) Successful in 5s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 23s
Release / image (push) Successful in 55s
Release / scan-image (push) Successful in 2s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as 9bf4c92 and 2b396d2, so a
tree heading for v0.28.0 doesn't say 0.27.0.
2026-09-26 09:19:35 +02:00
Niklas Ye 9d1df2b611 Show week numbers on the rota, and assign a whole week from them
The rota grid starts each row with its ISO week number, and for an owner
the number is a button: one tap opens a sheet for that week, showing who
holds each of its seven days, and puts one person on all of them. A rota
is usually handed out by the week, and seven taps on seven days was the
only way to do it short of the range form.

ISO 8601 numbering, because the grid already runs Monday to Sunday: week
1 is the one holding the year's first Thursday, which is taken from the
Thursday of the row so the year boundaries come out right (2025-12-29 is
week 1 of 2026, 2020-12-31 is week 53).

Days already past are left alone. Who was on call last Tuesday is a fact,
and "the whole week" should not rewrite it, so a half-elapsed week covers
the days still to come and the sheet says so; a week that is entirely
over has nothing to assign. By default the assignment replaces whoever
holds those days, as the day sheet does and the sheet states, and a
checkbox limits it to the days nobody has yet. The overhang into the
neighbouring month is part of the same week and is included.

Web UI only: the existing schedule endpoint already takes a list of
dates and a replace flag, so nothing changed on the server and there is
nothing to mirror in terdut-tui.
2026-09-26 09:19:35 +02:00
Niklas Ye 9bf4c92bfe Set the chart's placeholder version to 0.27.0
CI / chart (push) Successful in 1s
CI / security (push) Successful in 14s
CI / test (push) Successful in 2m59s
Release / test (push) Successful in 8s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 19s
Release / image (push) Successful in 54s
Release / scan-image (push) Successful in 27s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as 2b396d2 and dc92f51, so a
tree heading for v0.27.0 doesn't say 0.26.0.
2026-09-26 09:04:54 +02:00
Niklas Ye e616c82646 Show team members as a list, with who is on call and who cannot be paged
Team -> Members was a two-column table and an inline add form. It is now
a table in the style of Switches, Sources and Escalation: a status
badge, the member, their role, their next rota day, when they were last
active and when they joined. Adding a member and changing a role moved
into sheets, and removing one asks first.

The badge is the one that matters at 03:00: On call if the rota has them
today, Reachable if they have an ntfy topic, and Can't be paged when a
page to them would go nowhere -- no topic, or a disabled account -- with
the reason under their name. Not being pageable wins over being on call,
since an on-call person nobody can reach is the case worth seeing before
an incident finds it. The rules are the notifier's own. The topic itself
is never in the response, only whether one is set.

Last active is the newer of a member's newest session and API-key use,
and is shown to every member of the team like the rest of the list.
Rota days are UTC dates, and the page formats them as such so a day
cannot show up as the one before.

Removing a member leaves the rota days already assigned to them alone,
which the confirm says, so they are reassigned from the Rota tab rather
than silently dropped.

Demoting the last owner is now refused with 409, as removing them
already was: it was the same outcome by another route, a team with
nobody who can edit it.

API: GET /members gains status, on_call, next_shift, pageable, problem
and last_active_at; additive, no migration, and terdut-tui needs
nothing. POST /members answers 409 for the last-owner demotion.
2026-09-26 09:04:54 +02:00
Niklas Ye 2b396d22d6 Set the chart's placeholder version to 0.26.0
CI / chart (push) Successful in 1s
CI / security (push) Successful in 14s
CI / test (push) Successful in 2m56s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 22s
Release / image (push) Successful in 57s
Release / scan-image (push) Successful in 4s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as dc92f51 and e8d45f9, so a
tree heading for v0.26.0 doesn't say 0.25.0.
2026-09-26 08:28:24 +02:00
Niklas Ye 1f1faa437c Show the escalation ladder as a list, with who it would page and where it is
Team -> Escalation was the draft form on the page, which showed the
ladder only as inputs. It is now a table in the style of Switches and
Sources: a row per level with a status badge, who it pages, the wait
before the next level, and the open incidents currently waiting on it.
Below it, the repeat count, the fallback topic and when the ladder last
escalated (linking the incident). The editor moved into an "Edit ladder"
sheet, so a poll of the page underneath can no longer throw away half an
edit, and the page-level draft state went with it.

Targets are resolved to who they mean today, and the badge says what
would actually happen: Ready, Escalating (an unanswered incident has
climbed to level 2 or higher), or Pages nobody. The last is the one worth
seeing before an incident finds it: an empty rota, a person with no ntfy
topic or a disabled account each make a rung a silence with a number on
it, and the target says which. The rules are pageLevel's own, so the
page cannot promise a page the notifier would skip.

"Last escalated" comes from the escalated timeline events that already
exist, so there is no migration. Acknowledging or resolving takes an
incident off the ladder, so Escalating clears then while the history
stays.

API: GET /escalation gains status and waiting per level, username,
reachable and problem per target, and last_escalated_at and
last_escalated_incident_id. Output only and additive; PUT is unchanged
and terdut-tui needs nothing.
2026-09-26 08:28:24 +02:00
Niklas Ye dc92f51cf8 Set the chart's placeholder version to 0.25.0
CI / chart (push) Successful in 1s
CI / security (push) Successful in 31s
CI / test (push) Successful in 3m20s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 20s
Release / image (push) Successful in 1m2s
Release / scan-image (push) Successful in 2s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as e8d45f9 and 3ee8583, so a
tree heading for v0.25.0 doesn't say 0.24.0.
2026-09-26 07:52:07 +02:00
Niklas Ye d675f8ec9b List alert sources with their status and last arrival on Team -> Sources
Like Team -> Switches, the page is now a table: a status badge (Active
if the key posted within a day, Quiet if it has but not lately, Never
used), when it last posted a webhook, when an alert last arrived on it,
how many distinct alerts it refreshed in the last 24 hours, and when it
was created. Adding a source moved into a "New source" sheet, and owners
can rename one from its row.

"Last alert" and the count needed alerts to remember which source they
came in on, which they never did, so migration 010 adds
alerts.integration_id and every accepted payload stamps it. Last sender
wins when two sources post the same fingerprint. It is not backfilled: a
NULL says "before this was recorded" rather than guessing, and it heals
by itself as Alertmanager re-sends each alert every repeat_interval.
Revoking a source keeps its alerts, unattributed.

Last webhook and last alert are separate on purpose: a payload with
nothing usable in it stamps the first and not the second. The Quiet
threshold is a fixed day, a colour and not an alarm, since silence that
should page is what dead man's switches are for.

The counts are indexed subqueries (alerts_integration_idx) rather than a
join, which would read every alert a source ever delivered.

API: the integrations list gains status, last_alert_at and alerts_24h,
and PATCH /api/teams/{id}/integrations/{id} renames. Both are additive;
terdut-tui needs nothing.
2026-09-26 07:52:07 +02:00
Niklas Ye e8d45f9d3d Set the chart's placeholder version to 0.24.0
CI / chart (push) Successful in 1s
CI / security (push) Successful in 16s
CI / test (push) Successful in 2m55s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 24s
Release / image (push) Successful in 1m0s
Release / scan-image (push) Successful in 4s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as 3ee8583 and d2cdcc9, so a
tree heading for v0.24.0 doesn't say 0.23.0.
2026-09-25 23:40:55 +02:00
Niklas Ye f3918b863c List dead man's switches with their status on Team -> Switches
The page was a bare form: it did not say which switches existed or
whether they were alive. It now lists them, each with a Healthy, Dead or
Dormant badge, when its heartbeat was last heard and when it last opened
an incident (linked while that incident is open). A matcher that several
clusters satisfy is broken down per cluster, since a live cluster must
not hide a dead one. The form moved into a "New switch" sheet, and each
row has a Remove with a confirm.

That needed a switch to be a thing, so switches are rows now
(migration 009) with their own name, matcher, timeout and severity,
instead of one string with one team-wide timeout in deadman_configs.
Existing configuration is split into one row per matcher; a team whose
timeout was zero simply has none. The sweeper and the status endpoint
share one death rule (deadmanAlert.dead), so the page cannot disagree
with the pager. Incident group keys are unchanged, so incidents that
are open across the upgrade keep working.

The environment defaults (TERDUT_DEADMAN_*) are seeded into teams once
per install, recorded in settings, so a team that deletes its last
switch does not get it back on the next restart. Installs that already
had per-team rows are marked as seeded by the migration.

Removing a switch stops the watching but leaves an incident it already
opened open until someone resolves it.

API: GET/PUT /api/teams/{id}/deadman are replaced by
GET/POST /deadman/switches and DELETE /deadman/switches/{switchID}.
terdut-tui does not call them, so nothing to mirror there.
2026-09-25 23:40:55 +02:00
Niklas Ye 3ee8583f6f Set the chart's placeholder version to 0.23.0
CI / chart (push) Successful in 1s
CI / security (push) Successful in 14s
CI / test (push) Successful in 2m35s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 21s
Release / image (push) Successful in 57s
Release / scan-image (push) Successful in 2s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as d2cdcc9 and 71d7e18, so a
tree heading for v0.23.0 doesn't say 0.22.1.
2026-09-25 19:12:33 +02:00
Niklas Ye 591d5b8df0 Copy an incident to the clipboard as Markdown
A button in the incident header (also `y`, and "Copy incident" in the
more menu) puts everything the page knows on the clipboard, for pasting
into a chat or an agent prompt with no integration involved.

The text carries the facts, every alert with all its labels and
annotations (the page only shows summary or description), the timeline
with notes in full, and the "Seen before" resolution notes. Times are
ISO 8601 and users are named rather than "you", since relative and
first-person wording is ambiguous once pasted elsewhere.

The async clipboard API needs a secure context and this server is often
reached over plain HTTP, so it falls back to execCommand.

Web UI only: no endpoint or JSON shape changed, so nothing to mirror in
terdut-tui.
2026-09-25 19:12:33 +02:00
Niklas Ye d2cdcc9776 Set the chart's placeholder version to 0.22.1
CI / chart (push) Successful in 1s
CI / security (push) Successful in 16s
CI / test (push) Successful in 2m45s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 21s
Release / image (push) Successful in 59s
Release / scan-image (push) Successful in 2s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as 71d7e18 and 734cd9c, so a
tree heading for v0.22.1 doesn't say 0.22.0.

Claude-Session: https://claude.ai/code/session_01MMados3BD1oSjevHxbmVqU
2026-09-25 17:25:17 +02:00
Niklas Ye 8b2789b9b2 Let the filter chips wrap in the desktop incident list
The list pane is 340-420px wide and its chip row scrolled sideways with the
scrollbar hidden. That works by swipe on a phone, but a mouse has nothing to
grab, so Archived (the last chip) could not be reached on a wide screen. In
the desktop layout the row now wraps instead, and the divider between the
status and team chips is hidden there, since it would sit mid-line.

Phones keep the sideways scroll: the rule is inside the min-width: 900px
block.

Claude-Session: https://claude.ai/code/session_01MMados3BD1oSjevHxbmVqU
2026-09-25 17:25:17 +02:00
Niklas Ye 71d7e1853a Set the chart's placeholder version to 0.22.0
CI / chart (push) Successful in 1s
CI / security (push) Successful in 14s
CI / test (push) Successful in 2m41s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 18s
Release / image (push) Successful in 56s
Release / scan-image (push) Successful in 5s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as 734cd9c and 43f0044, so a
tree heading for v0.22.0 doesn't say 0.21.0.

Claude-Session: https://claude.ai/code/session_01MMados3BD1oSjevHxbmVqU
2026-09-25 16:55:12 +02:00
Niklas Ye 60ebb75cd2 Show notes from similar earlier incidents
Each incident gets a signature: the alert name plus the group labels that
say what is broken, minus the ones that only say where it ran (instance,
pod, container, ...). GET /api/incidents/{id}/similar returns resolved
incidents in the same team with the same signature that have notes.

Notes can be marked as the resolution note, "what fixed it", either with a
resolution field on resolve or pinned on a note. Those lead the similar
list, show on the incident page as "Seen before", and the triggered
notification carries the latest one.

Claude-Session: https://claude.ai/code/session_01MMados3BD1oSjevHxbmVqU
2026-09-25 15:42:25 +02:00
Niklas Ye 734cd9c5fd Set the chart's placeholder version to 0.21.0
CI / chart (push) Successful in 2s
CI / security (push) Successful in 16s
CI / test (push) Successful in 2m52s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 3s
Release / binaries (push) Successful in 18s
Release / image (push) Successful in 58s
Release / scan-image (push) Successful in 25s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as 43f0044 and e77f04b, so a
tree heading for v0.21.0 doesn't say 0.20.1.
2026-09-25 13:02:08 +02:00
Niklas Ye 423ed9b3a3 Add a Stats page to the web UI
Statistics used to live only in terdut-tui; the account page said so.
The page shows the same figures as the TUI's Stats tab -- incident
counts, MTTA and MTTR, top alerts, and alert frequency by hour (UTC) and
by day of week -- and adds a range picker (Today, 7d, 30d, 90d, All)
that the TUI does not have. The ranges are day-granular because the
server reads from/to as whole UTC dates, so there is no 24h chip.

No server change: the page uses the existing /api/stats/* endpoints,
which already scope to the caller's teams. Charts are inline SVG and
plain elements sized from script, because the CSP forbids inline styles,
inline scripts and CDN libraries.

Removes the "statistics are in terdut-tui" notes from the account page
and the README.
2026-09-25 13:02:08 +02:00
Niklas Ye 43f004499b Set the chart's placeholder version to 0.20.1
CI / chart (push) Successful in 1s
CI / security (push) Successful in 13s
CI / test (push) Successful in 2m30s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 15s
Release / image (push) Successful in 55s
Release / scan-image (push) Successful in 2s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as e77f04b and 429d5fd, so a
tree heading for v0.20.1 doesn't say 0.20.0.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
2026-09-22 22:42:13 +02:00
Niklas Ye 559be6de6e Retry the first database ping instead of dying on it
Every start crashed once or twice before going healthy: kube-router
enforces this namespace's NetworkPolicy per-node, reacting to the new
pod's creation event, and the app's first connection attempt can
reach Postgres's node before that node's allow-set has been updated
with the new pod's IP. The result is "connection refused" -- an
active reject, not a timeout, which is how it was told apart from
Postgres itself not being ready (it had been up for two days in the
run that was diagnosed).

That race resolves within several seconds in practice, so Open now
retries the ping up to five times, two seconds apart, logging each
failure, before giving up with the same wrapped error as before.
Nothing else about Open's behaviour changed: a genuinely absent
database still fails, just after ~8s instead of immediately.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
2026-09-22 22:42:05 +02:00
Niklas Ye e77f04b55e Set the chart's placeholder version to 0.20.0
CI / chart (push) Successful in 2s
CI / security (push) Successful in 20s
CI / test (push) Successful in 2m35s
Release / test (push) Successful in 4s
Release / chart (push) Successful in 2s
Release / binaries (push) Successful in 18s
Release / image (push) Successful in 55s
Release / scan-image (push) Successful in 2s
Cosmetic: make helm-package sets the published version and appVersion
from the tag, so these two fields decide nothing (see the comment
above them). Kept in step anyway, same as 429d5fd and 6a03698, so a
tree heading for v0.20.0 doesn't say 0.19.0.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
2026-09-22 21:47:25 +02:00
Niklas Ye e536fdd2c0 Replace the mobile tab bar with a hamburger menu
Six tabs (Queue, On-call, Alerts, Team, Admin, Account) had already
outgrown the bottom bar once: 56b8191 let it auto-size its columns to
fit however many there were, but on a phone that only left each tab
55-65px wide. Squeezing further wasn't an option, so the bar is gone
on mobile and a hamburger button in the topbar opens a menu instead.

The menu reuses the sheet + menu-item pattern already used for the
snooze and assign-to actions in incident.js, rather than a new overlay
component. It lists the same sections the sidebar does, including
hiding Admin for non-admins, and the triggered-incident badge that
used to sit on the Queue tab icon now shows on the hamburger button.

Desktop (>=900px) is untouched: the sidebar is the same markup, CSS
alone hides it below 900px and shows it above, so nothing there
changed behaviourally.

Claude-Session: https://claude.ai/code/session_01RHPj4ggeFdEjKKfm4SHbD7
2026-09-22 21:47:15 +02:00
34 changed files with 3142 additions and 554 deletions
+35 -24
View File
@@ -48,11 +48,12 @@ curl -H "Authorization: Bearer $KEY" http://localhost:8080/api/users
The server serves a web UI at `/`: the incident queue, each incident's alerts
and timeline with every action (acknowledge, assign, snooze, note, resolve,
archive), who is on call, the alert feed, and an *Account* tab for your own
password and the ntfy topic your pages go to. It is built for a phone first. On a phone it has a bottom tab bar and a sticky action
bar, it follows the system's dark mode, and it can be added to the home screen.
From 900px wide it switches to a sidebar with the queue and the incident side by
side. Statistics remain in
[terdut-tui](https://github.com/yeniklas/terdut-tui) for now.
password and the ntfy topic your pages go to. It is built for a phone first. On a phone
it navigates through a hamburger menu and has a sticky action bar, it follows the
system's dark mode, and it can be added to the home screen. From 900px wide it switches
to a sidebar with the queue and the incident side by side. The Stats page shows
incident counts, MTTA and MTTR, and alert frequency by name, hour and day over a
chosen range.
You sign in with a username and password. Users have no password until one is
set, and a user without one can only use API keys:
@@ -512,19 +513,27 @@ nothing unless something downstream notices it stop. That is what
`TERDUT_DEADMAN_MATCHERS` defaults to.
**Switches belong to a team**, which decides which of its own alerts are
heartbeats and how long a silence has to last. An owner sets them through
`PUT /api/teams/{teamID}/deadman`; a missed heartbeat opens an incident in the
team whose integration received it.
heartbeats and how long a silence has to last. Each **switch** is a row of its
own — a name, one matcher, a timeout and a severity — so switches in one team
can have different deadlines. An owner adds and removes them on **Team →
Switches**, which lists each with a status (**healthy**, **dead**, or
**dormant** until its first heartbeat), when it was last heard from, and when it
last opened an incident; a matcher that several clusters satisfy is broken down
per cluster. The API is `POST`/`DELETE /api/teams/{teamID}/deadman/switches`. A
missed heartbeat opens an incident in the team whose integration received it.
Removing a switch stops the watching; an incident it already opened stays open
until somebody resolves it.
The environment variables are the starting point, not the setting: at startup
every team **without** a configuration of its own is given one from them, and an
owner's later edit is never overwritten by a redeploy. A team created after
that starts watching nothing until its owner says otherwise — inheriting an
install-wide heartbeat would page a new team about a source it has never heard
of.
The environment variables are the starting point, not the setting: the **first**
time the server starts, every team is given a switch per default matcher from
them, once. After that a team's switches are its own — an owner's edit or
deletion is never put back by a redeploy. A team created later starts watching
nothing until its owner says otherwise — inheriting an install-wide heartbeat
would page a new team about a source it has never heard of.
A matcher is a set of exact label conditions, one of which must be the
`alertname`, in the same format the environment variable uses:
`alertname`, in the format the environment variable uses (one matcher per switch; the
variable takes several, separated by `;`):
```
alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat
@@ -706,19 +715,21 @@ administrator who is not in the team gets the same `404` as anybody else.
| `POST` | `/api/teams` | any | Create a team `{"name"}`; the creator becomes its first owner |
| `PUT` | `/api/teams/{teamID}` | **owner** | Rename it `{"name"}`. `409` if the name is taken |
| `DELETE` | `/api/teams/{teamID}` | **owner** | Delete a team and everything under it. `409` while it has open incidents |
| `GET` | `/api/teams/{teamID}/members` | member | Who is in the team |
| `POST` | `/api/teams/{teamID}/members` | **owner** | Add a member, or change their role `{"user_id","role"}` |
| `GET` | `/api/teams/{teamID}/members` | member | Who is in the team, with `status` (`oncall` if the rota has them today, `unpageable` when a page to them would go nowhere — even if they are on call — else `reachable`), `on_call`, `next_shift` (first rota day after today), `pageable` and `problem` (`has no ntfy topic` / `account is disabled`; never the topic itself) and `last_active_at` (their newest session or API-key use). Every member sees the same list |
| `POST` | `/api/teams/{teamID}/members` | **owner** | Add a member, or change their role `{"user_id","role"}`. `409` when it would demote the last owner |
| `DELETE` | `/api/teams/{teamID}/members/{userID}` | **owner** | Remove a member. `409` for the last owner |
| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys |
| `GET` | `/api/teams/{teamID}/integrations` | member | List integrations. Never returns keys. Each carries `status` (`active` if its key posted within 24h, `quiet` if it has but not lately, `never`), `last_used_at` (last webhook, usable or not), `last_alert_at` (when an alert last arrived on it) and `alerts_24h` (distinct alerts it refreshed in the last day). Alerts delivered before the source was recorded (migration 010) have none, so the last two fill in as Alertmanager re-sends them |
| `PATCH` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Rename `{"name"}`. The key does not change |
| `POST` | `/api/teams/{teamID}/integrations` | **owner** | Mint an integration `{"name","kind"}` — key and URL shown once |
| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration |
| `DELETE` | `/api/teams/{teamID}/integrations/{integrationID}` | **owner** | Revoke an integration. Alerts it delivered stay, unattributed |
| `GET` | `/api/teams/{teamID}/invites` | **owner** | The team's invite links, with their uses and expiry. Never the tokens |
| `POST` | `/api/teams/{teamID}/invites` | **owner** | Mint one `{"role","max_uses"}` — the full URL is returned once |
| `DELETE` | `/api/teams/{teamID}/invites/{inviteID}` | **owner** | Revoke a link before it expires |
| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[]}`. Empty levels means the team has none |
| `GET` | `/api/teams/{teamID}/escalation` | member | The team's [escalation ladder](#escalation) `{repeat_count, fallback_topic, levels[], last_escalated_at?, last_escalated_incident_id?}`. Empty levels means the team has none. Each level also carries `status` (`ready`, `escalating` when an unanswered incident has climbed to it, `unreachable` when nobody on it could be woken), `waiting` (ids of the open incidents on it) and, per target, `username` (who it means today — the person on call, for a rota target), `reachable` and `problem`. The extra fields are output only; `PUT` takes the plain shape |
| `PUT` | `/api/teams/{teamID}/escalation` | **owner** | Replace it wholesale. `400` for a level with no targets or no timeout — a rung that pages nobody is a silence with a number on it |
| `GET` | `/api/teams/{teamID}/deadman` | member | The team's [dead man's switch](#dead-mans-switch) configuration `{matchers, timeout_seconds, severity}` |
| `PUT` | `/api/teams/{teamID}/deadman` | **owner** | Replace it. `400` when no matcher names an `alertname`, because a switch that silently watches nothing is the failure this feature exists to prevent |
| `GET` | `/api/teams/{teamID}/deadman/switches` | member | The team's [dead man's switches](#dead-mans-switch), each `{id, name, matcher, timeout_seconds, severity, status, last_heartbeat_at, last_triggered_at, open_incident_id, sources[]}`. `status` is `healthy`, `dead` or `dormant`; `sources` has one entry per heartbeat fingerprint. Empty when the team watches nothing |
| `POST` | `/api/teams/{teamID}/deadman/switches` | **owner** | Add one: `{name?, matcher, timeout_seconds, severity?}`. `400` when the matcher names no `alertname` or holds several, or the timeout is not positive — a switch that silently watches nothing is the failure this feature exists to prevent |
| `DELETE` | `/api/teams/{teamID}/deadman/switches/{switchID}` | **owner** | Stop watching. An incident it opened stays open. `404` for a switch of another team |
### Notifications
@@ -963,8 +974,8 @@ What changes, and will need attention:
**Dead man's switches moved too.** `TERDUT_DEADMAN_MATCHERS`, `_TIMEOUT` and
`_SEVERITY` are no longer the setting; they are the default each existing team
is seeded with at startup, after which an owner edits them per team through
`PUT /api/teams/{teamID}/deadman` and a redeploy never overwrites that.
is seeded with at startup, after which an owner manages them per team through
`/api/teams/{teamID}/deadman/switches` and a redeploy never overwrites that.
Nothing else about an incident changes, and incidents never move between teams:
an alert belongs to whichever team's key it arrived on.
+2 -2
View File
@@ -15,5 +15,5 @@ type: application
# appVersion and image.tag in values.yaml no longer agree, and that is not an oversight:
# image.tag stays "latest", which is what a local install actually pulls. appVersion is
# metadata and drives nothing.
version: 0.19.0
appVersion: "v0.19.0"
version: 0.28.1
appVersion: "v0.28.1"
+19 -14
View File
@@ -78,7 +78,7 @@ type ingested struct {
// post, and which team the alerts belong to.
func handleIntegrationWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, err := teamIDForKey(r.Context(), db, chi.URLParam(r, "key"))
src, err := sourceForKey(r.Context(), db, chi.URLParam(r, "key"))
if err != nil {
if errors.Is(err, errUnknownIntegration) {
// 401 and not 404: the path is real, the key is not, and a
@@ -90,11 +90,12 @@ func handleIntegrationWebhook(db *sql.DB, notify NotifyConfig) http.HandlerFunc
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
receiveWebhook(w, r, db, notify, teamID)
receiveWebhook(w, r, db, notify, src)
}
}
func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify NotifyConfig, teamID int64) {
func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify NotifyConfig, src alertSource) {
teamID := src.teamID
var payload amPayload
if err := decodeJSON(r, &payload); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid payload"))
@@ -104,7 +105,7 @@ func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify N
// Alertmanager retries anything that is not 2xx, and a retry of a payload
// we failed to store is more useful than an error it cannot act on — so
// failures are logged, not surfaced.
if err := ingest(r.Context(), db, notify, teamID, payload); err != nil {
if err := ingest(r.Context(), db, notify, src, payload); err != nil {
log.Printf("webhook ingest (team %d, group %q): %v", teamID, payload.GroupKey, err)
}
@@ -114,7 +115,8 @@ func receiveWebhook(w http.ResponseWriter, r *http.Request, db *sql.DB, notify N
// ingest stores a payload's alerts and reconciles the incident for its group.
// The whole payload is one transaction: an incident that opened but whose alerts
// failed to link would be a work item nobody could act on.
func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64, payload amPayload) error {
func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, src alertSource, payload amPayload) error {
teamID := src.teamID
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
@@ -124,12 +126,12 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64,
// Which arriving alerts are heartbeats is the team's own answer, read
// inside the transaction so an owner editing it mid-payload cannot split
// one webhook across two interpretations.
deadman, err := deadmanConfigForTeam(ctx, tx, teamID)
deadman, err := deadmanSetForTeam(ctx, tx, teamID)
if err != nil {
return err
}
accepted, err := upsertAlerts(ctx, tx, deadman, teamID, payload.Alerts)
accepted, err := upsertAlerts(ctx, tx, deadman, src, payload.Alerts)
if err != nil {
return err
}
@@ -186,7 +188,8 @@ func ingest(ctx context.Context, db *sql.DB, notify NotifyConfig, teamID int64,
// upsertAlerts stores each alert of a payload and reports what changed. Payloads
// the ordering guard rejected are left out entirely.
func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID int64, alerts []amAlert) ([]ingested, error) {
func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman deadmanSet, src alertSource, alerts []amAlert) ([]ingested, error) {
teamID := src.teamID
now := time.Now().Unix()
accepted := make([]ingested, 0, len(alerts))
@@ -242,8 +245,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID
if _, err := tx.ExecContext(ctx, `
INSERT INTO alerts
(team_id, fingerprint, name, status, labels, annotations, starts_at, ends_at,
generator_url, received_at, resolution_source)
VALUES ($1, $2, $3, $4, $5::jsonb, $6::jsonb, $7, $8, $9, $10, $11)
generator_url, received_at, resolution_source, integration_id)
VALUES ($1, $2, $3, $4, $5::jsonb, $6::jsonb, $7, $8, $9, $10, $11, $12)
ON CONFLICT (team_id, fingerprint) DO UPDATE SET
status = excluded.status,
labels = excluded.labels,
@@ -257,6 +260,8 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID
-- is a breaking API change — see models.Alert.ReceivedAt.
received_at = excluded.received_at,
resolution_source = excluded.resolution_source,
-- Last sender wins; see migration 010.
integration_id = excluded.integration_id,
-- A re-fire makes the alert current again, so it leaves the archive.
archived_at = CASE WHEN excluded.status = 'firing'
THEN NULL ELSE alerts.archived_at END
@@ -267,7 +272,7 @@ func upsertAlerts(ctx context.Context, tx *sql.Tx, deadman DeadmanConfig, teamID
teamID, a.Fingerprint, name, a.Status,
string(labelsJSON), string(annotationsJSON),
a.StartsAt.Unix(), endsAtUnix,
a.GeneratorURL, now, resolutionSource,
a.GeneratorURL, now, resolutionSource, src.integrationID,
); err != nil {
return nil, err
}
@@ -384,10 +389,10 @@ func openIncident(ctx context.Context, q querier, notify NotifyConfig, teamID in
var id int64
err = q.QueryRowContext(ctx, `
INSERT INTO incidents (team_id, group_key, title, group_labels, status, severity, triggered_at, assigned_to)
VALUES ($1, $2, $3, $4::jsonb, 'triggered', $5, $6, $7)
INSERT INTO incidents (team_id, group_key, title, group_labels, signature, status, severity, triggered_at, assigned_to)
VALUES ($1, $2, $3, $4::jsonb, $5, 'triggered', $6, $7, $8)
RETURNING id`,
teamID, groupKey, title, string(labelsJSON), severity,
teamID, groupKey, title, string(labelsJSON), incidentSignature(groupLabels, title), severity,
time.Now().Unix(), onCall).Scan(&id)
if err != nil {
return 0, err
+8 -10
View File
@@ -88,27 +88,25 @@ func newDeadmanTS(t *testing.T, deadman api.DeadmanConfig, notify ...api.NotifyC
return s
}
// setTeamDeadman configures the default team's switches over the API, rendering
// the matchers back into the string form the endpoint takes.
// setTeamDeadman gives the default team one switch per configured matcher, over
// the API, the way an owner would add them.
func setTeamDeadman(t *testing.T, s *ts, cfg api.DeadmanConfig) {
t.Helper()
matchers := make([]string, 0, len(cfg.Matchers))
for _, m := range cfg.Matchers {
parts := []string{"alertname=" + m.Name}
for k, v := range m.Labels {
parts = append(parts, k+"="+v)
}
sort.Strings(parts[1:])
matchers = append(matchers, strings.Join(parts, ","))
}
resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{
"matchers": strings.Join(matchers, "; "),
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", map[string]any{
"matcher": strings.Join(parts, ","),
"timeout_seconds": int64(cfg.Timeout.Seconds()),
"severity": cfg.Severity,
})
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("configure the team's dead man's switches: %d", resp.StatusCode)
resp.Body.Close()
if resp.StatusCode != http.StatusCreated {
t.Fatalf("add a dead man's switch: %d", resp.StatusCode)
}
}
}
+350 -170
View File
@@ -4,6 +4,8 @@ import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"log"
"sort"
"strings"
@@ -39,6 +41,17 @@ func (m DeadmanMatcher) String() string {
return m.Name + " (" + strings.Join(parts, ", ") + ")"
}
// config renders the matcher in the form parseDeadmanMatcher reads, which is
// what a switch row stores: `alertname=Watchdog,cluster=prod`.
func (m DeadmanMatcher) config() string {
parts := make([]string, 0, len(m.Labels))
for k, v := range m.Labels {
parts = append(parts, k+"="+v)
}
sort.Strings(parts)
return strings.Join(append([]string{"alertname=" + m.Name}, parts...), ",")
}
// matches reports whether an alert's labels satisfy every condition.
func (m DeadmanMatcher) matches(labels map[string]string) bool {
if labels["alertname"] != m.Name {
@@ -52,12 +65,10 @@ func (m DeadmanMatcher) matches(labels map[string]string) bool {
return true
}
// DeadmanConfig inverts the handling of the alerts it matches: receiving one
// opens nothing, and the absence of one opens an incident.
//
// The unit of monitoring is the fingerprint, not the matcher — two clusters
// sending the same heartbeat alertname are two independent switches, so one
// healthy cluster cannot mask a dead one.
// DeadmanConfig is the server-wide default a team's switches are seeded from:
// the environment's matchers, timeout and severity. Switches themselves are rows
// of a team's own — see DeadmanSwitch — and this is only how a fresh install
// starts out.
type DeadmanConfig struct {
Matchers []DeadmanMatcher
@@ -76,41 +87,81 @@ type DeadmanConfig struct {
// enabled reports whether there is anything to watch.
func (c DeadmanConfig) enabled() bool { return c.Timeout > 0 && len(c.Matchers) > 0 }
// match returns the first matcher an alert satisfies.
func (c DeadmanConfig) match(labels map[string]string) (DeadmanMatcher, bool) {
if !c.enabled() {
return DeadmanMatcher{}, false
}
for _, m := range c.Matchers {
if m.matches(labels) {
return m, true
}
}
return DeadmanMatcher{}, false
// DeadmanSwitch inverts the handling of the alerts it matches: receiving one
// opens nothing, and the absence of one opens an incident.
//
// The unit of monitoring is the fingerprint, not the switch — two clusters
// sending the same heartbeat alertname are two independent heartbeats under one
// switch, so one healthy cluster cannot mask a dead one.
type DeadmanSwitch struct {
ID int64
Name string
Matcher DeadmanMatcher
// Timeout is how long a heartbeat may go unheard before it is declared dead.
Timeout time.Duration
// Severity is what the incident opens at.
Severity string
}
// isDeadman is match without the matcher, for the ingest path.
func (c DeadmanConfig) isDeadman(labels map[string]string) bool {
_, ok := c.match(labels)
// deadmanSet is one team's switches.
type deadmanSet []DeadmanSwitch
// match returns the first switch an alert satisfies.
func (d deadmanSet) match(labels map[string]string) (DeadmanSwitch, bool) {
for _, sw := range d {
if sw.Matcher.matches(labels) {
return sw, true
}
}
return DeadmanSwitch{}, false
}
// isDeadman is match without the switch, for the ingest path.
func (d deadmanSet) isDeadman(labels map[string]string) bool {
_, ok := d.match(labels)
return ok
}
// names lists the distinct alertnames worth loading from the database.
func (c DeadmanConfig) names() []string {
func (d deadmanSet) names() []string {
seen := map[string]bool{}
out := make([]string, 0, len(c.Matchers))
for _, m := range c.Matchers {
if !seen[m.Name] {
seen[m.Name] = true
out = append(out, m.Name)
out := make([]string, 0, len(d))
for _, sw := range d {
if !seen[sw.Matcher.Name] {
seen[sw.Matcher.Name] = true
out = append(out, sw.Matcher.Name)
}
}
return out
}
// parseDeadmanMatcher reads one matcher from its configured form: "," separates
// the conditions and "=" is exact label equality — `alertname=Watchdog,cluster=prod`.
// The error says what is wrong with it, in words a form can show.
func parseDeadmanMatcher(entry string) (DeadmanMatcher, error) {
m := DeadmanMatcher{Labels: map[string]string{}}
for _, cond := range strings.Split(strings.TrimSpace(entry), ",") {
k, v, ok := strings.Cut(cond, "=")
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
if !ok || k == "" || v == "" {
return DeadmanMatcher{}, fmt.Errorf("%q is not label=value", strings.TrimSpace(cond))
}
if k == "alertname" {
m.Name = v
continue
}
m.Labels[k] = v
}
if m.Name == "" {
return DeadmanMatcher{}, errors.New("no alertname condition")
}
return m, nil
}
// ParseDeadmanConfig reads the matcher list from its configured form:
// ";" separates matchers, "," separates the conditions within one, and "=" is
// exact label equality — `alertname=Watchdog,cluster=prod; alertname=Heartbeat`.
// ";" separates matchers, and each is parsed as parseDeadmanMatcher does.
//
// A malformed or alertname-less entry is dropped rather than fatal, following
// config.duration's rule that one bad tuning knob should not take the server
@@ -125,28 +176,9 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string)
if entry == "" {
continue
}
m := DeadmanMatcher{Labels: map[string]string{}}
malformed := false
for _, cond := range strings.Split(entry, ",") {
k, v, ok := strings.Cut(cond, "=")
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
if !ok || k == "" || v == "" {
log.Printf("deadman: ignoring matcher %q: %q is not label=value", entry, strings.TrimSpace(cond))
malformed = true
break
}
if k == "alertname" {
m.Name = v
continue
}
m.Labels[k] = v
}
if malformed {
continue
}
if m.Name == "" {
log.Printf("deadman: ignoring matcher %q: no alertname condition", entry)
m, err := parseDeadmanMatcher(entry)
if err != nil {
log.Printf("deadman: ignoring matcher %q: %v", entry, err)
continue
}
cfg.Matchers = append(cfg.Matchers, m)
@@ -162,23 +194,35 @@ func ParseDeadmanConfig(matchers string, timeout time.Duration, severity string)
for _, m := range cfg.Matchers {
rendered = append(rendered, m.String())
}
log.Printf("deadman: watching %s, timeout %s, severity %s",
log.Printf("deadman: default for new teams: %s, timeout %s, severity %s",
strings.Join(rendered, "; "), timeout, severity)
}
return cfg
}
// deadmanAlert is one switch: the alert row carrying its last heartbeat.
// deadmanAlert is one heartbeat: the alert row carrying its last sighting, and
// the switch that claimed it.
type deadmanAlert struct {
id int64
teamID int64
fingerprint string
labels map[string]string
matcher DeadmanMatcher
sw DeadmanSwitch
resolved bool
receivedAt int64
}
// dead is the one rule for a silent heartbeat, shared by the sweeper that pages
// on it and the status the Switches page shows, so the page cannot disagree
// with the pager.
//
// An explicit resolved from Alertmanager is a stronger death signal than mere
// absence: the sender is telling us the heartbeat stopped, so there is nothing
// left to wait out.
func (a deadmanAlert) dead(now time.Time) bool {
return a.resolved || a.receivedAt < now.Add(-a.sw.Timeout).Unix()
}
// groupKey is the switch's identity as an incident. Per fingerprint, so each
// source is tracked on its own.
func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerprint }
@@ -189,13 +233,13 @@ func (a deadmanAlert) groupKey() string { return deadmanGroupPrefix + a.fingerpr
// It returns the ids of the alerts it owns, because the generic staleness
// expiry must leave them alone — staleAfter and ends_at would otherwise resolve
// a heartbeat long before its own, much tighter, timeout ever fired.
// Each team is swept against its own configuration: its own matchers, its own
// timeout, its own severity. A team watching nothing is skipped entirely, which
// is most of them.
// Each team is swept against its own switches, each with its own matcher,
// timeout and severity. A team watching nothing is skipped entirely, which is
// most of them.
func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int64]bool {
owned := map[int64]bool{}
configs, err := deadmanConfigs(ctx, db)
configs, err := deadmanSets(ctx, db)
if err != nil {
log.Printf("deadman: load configs: %v", err)
return owned
@@ -203,39 +247,35 @@ func sweepDeadman(ctx context.Context, db *sql.DB, notify NotifyConfig) map[int6
now := time.Now()
for teamID, cfg := range configs {
switches, err := deadmanAlerts(ctx, db, teamID, cfg)
heartbeats, err := deadmanAlerts(ctx, db, teamID, cfg)
if err != nil {
log.Printf("deadman: load switches for team %d: %v", teamID, err)
log.Printf("deadman: load heartbeats for team %d: %v", teamID, err)
continue
}
cutoff := now.Add(-cfg.Timeout).Unix()
for _, sw := range switches {
owned[sw.id] = true
for _, hb := range heartbeats {
owned[hb.id] = true
// An explicit resolved from Alertmanager is a stronger death signal
// than mere absence: the sender is telling us the heartbeat
// stopped, so there is nothing left to wait out.
if sw.resolved || sw.receivedAt < cutoff {
if err := deadmanDied(ctx, db, cfg, notify, sw, now); err != nil {
log.Printf("deadman: open incident for %s: %v", sw.matcher.Name, err)
if hb.dead(now) {
if err := deadmanDied(ctx, db, notify, hb, now); err != nil {
log.Printf("deadman: open incident for %s: %v", hb.sw.Matcher.Name, err)
}
continue
}
if err := deadmanRecovered(ctx, db, sw); err != nil {
log.Printf("deadman: resolve incident for %s: %v", sw.matcher.Name, err)
if err := deadmanRecovered(ctx, db, hb); err != nil {
log.Printf("deadman: resolve incident for %s: %v", hb.sw.Matcher.Name, err)
}
}
}
return owned
}
// deadmanAlerts loads every alert row that a matcher claims. The candidate query
// deadmanAlerts loads every alert row that one of a team's switches claims. The candidate query
// is narrowed by alertname so it rides alerts_name_idx; the rest of the matching
// happens in Go, which keeps one implementation of the rules. The rows are read
// in full before the caller writes, so the writes do not run against an open
// cursor over the same table.
func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanConfig) ([]deadmanAlert, error) {
func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg deadmanSet) ([]deadmanAlert, error) {
names := cfg.names()
args := &sqlArgs{}
nameList := make([]any, len(names))
@@ -263,11 +303,11 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanCon
}
json.Unmarshal([]byte(labelsJSON), &a.labels) //nolint:errcheck
m, ok := cfg.match(a.labels)
sw, ok := cfg.match(a.labels)
if !ok {
continue
}
a.matcher = m
a.sw = sw
a.resolved = status == "resolved"
out = append(out, a)
}
@@ -284,16 +324,16 @@ func deadmanAlerts(ctx context.Context, db *sql.DB, teamID int64, cfg DeadmanCon
// incidentForGroup), and a source that is gone for good is a one-time page
// rather than a nag. Only a heartbeat that comes back and dies again earns a new
// incident.
func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify NotifyConfig, sw deadmanAlert, now time.Time) error {
func deadmanDied(ctx context.Context, db *sql.DB, notify NotifyConfig, hb deadmanAlert, now time.Time) error {
var lastTriggered, open int64
if err := db.QueryRowContext(ctx, `
SELECT COALESCE(MAX(triggered_at), 0),
COUNT(*) FILTER (WHERE resolved_at IS NULL)
FROM incidents WHERE team_id = $1 AND group_key = $2`,
sw.teamID, sw.groupKey()).Scan(&lastTriggered, &open); err != nil {
hb.teamID, hb.groupKey()).Scan(&lastTriggered, &open); err != nil {
return err
}
if open > 0 || sw.receivedAt <= lastTriggered {
if open > 0 || hb.receivedAt <= lastTriggered {
return nil
}
@@ -306,18 +346,18 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
// A heartbeat nobody has heard from is not firing, and saying otherwise in
// the alert list would be a lie. An Alertmanager-sourced resolution keeps its
// own source: it told us the truth first.
if !sw.resolved {
if !hb.resolved {
if _, err := tx.ExecContext(ctx, `
UPDATE alerts
SET status = 'resolved',
resolution_source = $1,
ends_at = COALESCE(ends_at, `+nowEpoch+`)
WHERE id = $2 AND status = 'firing'`, resolutionDeadman, sw.id); err != nil {
WHERE id = $2 AND status = 'firing'`, resolutionDeadman, hb.id); err != nil {
return err
}
}
severity := cfg.Severity
severity := hb.sw.Severity
var sev *string
if severity != "" {
sev = &severity
@@ -325,14 +365,14 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
// The incident opens in the team whose integration received the heartbeat:
// the switch belongs to whoever is watching that source, not to the install.
incidentID, err := openIncident(ctx, tx, notify, sw.teamID, sw.groupKey(),
"No heartbeat from "+sw.matcher.String(), sw.labels, sev)
incidentID, err := openIncident(ctx, tx, notify, hb.teamID, hb.groupKey(),
"No heartbeat from "+hb.sw.Matcher.String(), hb.labels, sev)
if err != nil {
return err
}
alertID := sw.id
detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(sw.receivedAt, 0))) + " ago"
alertID := hb.id
detail := "last heartbeat " + humanDuration(now.Sub(time.Unix(hb.receivedAt, 0))) + " ago"
if err := logEvent(ctx, tx, incidentID, evDeadmanSilent, nil, &alertID, &detail); err != nil {
return err
}
@@ -340,7 +380,7 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
if err := tx.Commit(); err != nil {
return err
}
log.Printf("deadman: %s went silent, opened incident %d", sw.matcher.String(), incidentID)
log.Printf("deadman: %s went silent, opened incident %d", hb.sw.Matcher.String(), incidentID)
return nil
}
@@ -350,12 +390,12 @@ func deadmanDied(ctx context.Context, db *sql.DB, cfg DeadmanConfig, notify Noti
// member alerts (linking the heartbeat would have the settled-incident cascade
// close it on the very same sweep that opened it), so the alert-driven cascade
// ignores it entirely and recovery is the only automatic way out.
func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error {
func deadmanRecovered(ctx context.Context, db *sql.DB, hb deadmanAlert) error {
var incidentID int64
switch err := db.QueryRowContext(ctx, `
SELECT id FROM incidents
WHERE team_id = $1 AND group_key = $2 AND resolved_at IS NULL`,
sw.teamID, sw.groupKey()).Scan(&incidentID); {
hb.teamID, hb.groupKey()).Scan(&incidentID); {
case err == sql.ErrNoRows:
return nil
case err != nil:
@@ -387,116 +427,256 @@ func deadmanRecovered(ctx context.Context, db *sql.DB, sw deadmanAlert) error {
if err := tx.Commit(); err != nil {
return err
}
log.Printf("deadman: %s is back, resolved incident %d", sw.matcher.String(), incidentID)
log.Printf("deadman: %s is back, resolved incident %d", hb.sw.Matcher.String(), incidentID)
return nil
}
// ---------------------------------------------------------------------------
// Per-team configuration
// A team's switches
// ---------------------------------------------------------------------------
// deadmanConfigForTeam reads one team's switches. A team with no row, or with
// nothing configured, gets a disabled config — which is the right answer rather
// than an error: most teams watch no heartbeat at all.
func deadmanConfigForTeam(ctx context.Context, q querier, teamID int64) (DeadmanConfig, error) {
var matchers, severity string
var timeout int64
err := q.QueryRowContext(ctx,
"SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1",
teamID).Scan(&matchers, &timeout, &severity)
if err == sql.ErrNoRows {
return DeadmanConfig{}, nil
}
if err != nil {
return DeadmanConfig{}, err
}
return parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity), nil
}
const deadmanSwitchColumns = "id, team_id, name, matcher, timeout_seconds, severity"
// deadmanConfigs reads every team's switches in one query, for the sweeper.
func deadmanConfigs(ctx context.Context, db *sql.DB) (map[int64]DeadmanConfig, error) {
rows, err := db.QueryContext(ctx,
"SELECT team_id, matchers, timeout_seconds, severity FROM deadman_configs")
if err != nil {
return nil, err
}
// scanDeadmanSwitches reads switch rows into per-team sets. A row whose matcher
// no longer parses is skipped rather than fatal: the API refuses to store one,
// so it can only mean a hand edit, and one bad row must not stop the others
// from being watched.
func scanDeadmanSwitches(rows *sql.Rows) (map[int64]deadmanSet, error) {
defer rows.Close()
out := map[int64]DeadmanConfig{}
out := map[int64]deadmanSet{}
for rows.Next() {
var sw DeadmanSwitch
var teamID, timeout int64
var matchers, severity string
if err := rows.Scan(&teamID, &matchers, &timeout, &severity); err != nil {
var matcher string
if err := rows.Scan(&sw.ID, &teamID, &sw.Name, &matcher, &timeout, &sw.Severity); err != nil {
return nil, err
}
cfg := parseDeadmanQuietly(matchers, time.Duration(timeout)*time.Second, severity)
if cfg.enabled() {
out[teamID] = cfg
m, err := parseDeadmanMatcher(matcher)
if err != nil {
log.Printf("deadman: switch %d has an unusable matcher %q: %v", sw.ID, matcher, err)
continue
}
sw.Matcher = m
sw.Timeout = time.Duration(timeout) * time.Second
out[teamID] = append(out[teamID], sw)
}
return out, rows.Err()
}
// SeedDeadmanConfigs gives every team without a row the server's environment
// configuration, so the install that upgrades into per-team switches keeps
// watching exactly what it was watching before.
//
// Idempotent, and never overwrites: once a team has a row it owns its own
// configuration, and a redeploy must not quietly put the environment's value
// back over an owner's edit.
//
// A team created after startup gets no row and therefore watches nothing until
// its owner says otherwise. That is deliberate: inheriting an install-wide
// heartbeat would page a new team about a source it has never heard of, and a
// switch nobody chose is the kind that gets muted rather than fixed.
func SeedDeadmanConfigs(ctx context.Context, db *sql.DB, cfg DeadmanConfig) error {
matchers := make([]string, 0, len(cfg.Matchers))
for _, m := range cfg.Matchers {
parts := []string{"alertname=" + m.Name}
for k, v := range m.Labels {
parts = append(parts, k+"="+v)
// deadmanSetForTeam reads one team's switches. A team with none gets an empty
// set — which is the right answer rather than an error: most teams watch no
// heartbeat at all.
func deadmanSetForTeam(ctx context.Context, q querier, teamID int64) (deadmanSet, error) {
rows, err := q.QueryContext(ctx,
"SELECT "+deadmanSwitchColumns+" FROM deadman_switches WHERE team_id = $1 ORDER BY id", teamID)
if err != nil {
return nil, err
}
sort.Strings(parts[1:])
matchers = append(matchers, strings.Join(parts, ","))
sets, err := scanDeadmanSwitches(rows)
return sets[teamID], err
}
_, err := db.ExecContext(ctx, `
INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity)
SELECT id, $1, $2, $3 FROM teams
ON CONFLICT (team_id) DO NOTHING`,
strings.Join(matchers, "; "), int64(cfg.Timeout.Seconds()), cfg.Severity)
// deadmanSets reads every team's switches in one query, for the sweeper.
func deadmanSets(ctx context.Context, db *sql.DB) (map[int64]deadmanSet, error) {
rows, err := db.QueryContext(ctx,
"SELECT "+deadmanSwitchColumns+" FROM deadman_switches ORDER BY id")
if err != nil {
return nil, err
}
return scanDeadmanSwitches(rows)
}
// deadmanSeededKey is the settings row that records the environment defaults
// were handed out. Without it, a team that deleted its last switch would get
// the default back on the next restart.
const deadmanSeededKey = "deadman_seeded"
// SeedDeadmanConfigs gives every team the server's environment defaults as
// switches, exactly once per install, so a fresh install watches Watchdog
// without anybody setting it up.
//
// Once seeded it never runs again: a team's switches are its own, and a redeploy
// must not quietly put the environment's value back over an owner's edit or
// deletion. Installs that upgraded from per-team configuration were already
// seeded, which migration 009 records.
//
// A team created after that gets none and watches nothing until its owner says
// otherwise. That is deliberate: inheriting an install-wide heartbeat would page
// a new team about a source it has never heard of, and a switch nobody chose is
// the kind that gets muted rather than fixed.
func SeedDeadmanConfigs(ctx context.Context, db *sql.DB, cfg DeadmanConfig) error {
if !cfg.enabled() {
return nil
}
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback() //nolint:errcheck
// parseDeadmanQuietly is ParseDeadmanConfig without the startup logging: a
// team's configuration is read on every sweep and every webhook, and logging it
// each time would bury everything else.
func parseDeadmanQuietly(matchers string, timeout time.Duration, severity string) DeadmanConfig {
cfg := DeadmanConfig{Timeout: timeout, Severity: severity}
for _, entry := range strings.Split(matchers, ";") {
entry = strings.TrimSpace(entry)
if entry == "" {
continue
res, err := tx.ExecContext(ctx,
"INSERT INTO settings (key, value) VALUES ($1, '1') ON CONFLICT (key) DO NOTHING",
deadmanSeededKey)
if err != nil {
return err
}
m := DeadmanMatcher{Labels: map[string]string{}}
malformed := false
for _, cond := range strings.Split(entry, ",") {
k, v, ok := strings.Cut(cond, "=")
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
if !ok || k == "" || v == "" {
malformed = true
break
if n, _ := res.RowsAffected(); n == 0 {
return nil
}
if k == "alertname" {
m.Name = v
continue
for _, m := range cfg.Matchers {
if _, err := tx.ExecContext(ctx, `
INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
SELECT id, $1, $1, $2, $3 FROM teams`,
m.config(), int64(cfg.Timeout.Seconds()), cfg.Severity); err != nil {
return err
}
m.Labels[k] = v
}
if malformed || m.Name == "" {
continue
return tx.Commit()
}
cfg.Matchers = append(cfg.Matchers, m)
// ---------------------------------------------------------------------------
// Status
// ---------------------------------------------------------------------------
const (
switchHealthy = "healthy"
switchDead = "dead"
switchDormant = "dormant"
)
// deadmanSource is one heartbeat under a switch: a fingerprint that matched.
type deadmanSource struct {
Fingerprint string `json:"fingerprint"`
Labels map[string]string `json:"labels"`
Status string `json:"status"`
LastHeartbeatAt time.Time `json:"last_heartbeat_at"`
LastTriggeredAt *time.Time `json:"last_triggered_at"`
IncidentID *int64 `json:"incident_id"`
}
return cfg
// deadmanSwitchStatus is a switch as the Switches page shows it.
type deadmanSwitchStatus struct {
ID int64 `json:"id"`
Name string `json:"name"`
Matcher string `json:"matcher"`
TimeoutSeconds int64 `json:"timeout_seconds"`
Severity string `json:"severity"`
// Status is dead when any source is, dormant when none has ever been heard
// from, healthy otherwise — a live cluster must not hide a dead one.
Status string `json:"status"`
LastHeartbeatAt *time.Time `json:"last_heartbeat_at"`
LastTriggeredAt *time.Time `json:"last_triggered_at"`
OpenIncidentID *int64 `json:"open_incident_id"`
Sources []deadmanSource `json:"sources"`
}
// deadmanStatuses reports every switch of a team with what its heartbeats are
// doing. The liveness verdict is deadmanAlert.dead, the sweeper's own.
func deadmanStatuses(ctx context.Context, db *sql.DB, teamID int64, set deadmanSet, now time.Time) ([]deadmanSwitchStatus, error) {
out := make([]deadmanSwitchStatus, 0, len(set))
if len(set) == 0 {
return out, nil
}
heartbeats, err := deadmanAlerts(ctx, db, teamID, set)
if err != nil {
return nil, err
}
// One query for every switch's incident history, keyed the way the sweeper
// keys it.
type history struct {
triggeredAt int64
openID int64
}
incidents := map[string]history{}
rows, err := db.QueryContext(ctx, `
SELECT group_key, MAX(triggered_at), COALESCE(MAX(id) FILTER (WHERE resolved_at IS NULL), 0)
FROM incidents
WHERE team_id = $1 AND group_key LIKE $2
GROUP BY group_key`, teamID, deadmanGroupPrefix+"%")
if err != nil {
return nil, err
}
defer rows.Close()
for rows.Next() {
var key string
var h history
if err := rows.Scan(&key, &h.triggeredAt, &h.openID); err != nil {
return nil, err
}
incidents[key] = h
}
if err := rows.Err(); err != nil {
return nil, err
}
bySwitch := map[int64][]deadmanAlert{}
for _, hb := range heartbeats {
bySwitch[hb.sw.ID] = append(bySwitch[hb.sw.ID], hb)
}
later := func(cur *time.Time, unix int64) *time.Time {
t := time.Unix(unix, 0).UTC()
if cur == nil || t.After(*cur) {
return &t
}
return cur
}
for _, sw := range set {
st := deadmanSwitchStatus{
ID: sw.ID, Name: sw.Name, Matcher: sw.Matcher.config(),
TimeoutSeconds: int64(sw.Timeout.Seconds()), Severity: sw.Severity,
Status: switchDormant, Sources: []deadmanSource{},
}
for _, hb := range bySwitch[sw.ID] {
src := deadmanSource{
Fingerprint: hb.fingerprint,
Labels: hb.labels,
Status: switchHealthy,
LastHeartbeatAt: time.Unix(hb.receivedAt, 0).UTC(),
}
if hb.dead(now) {
src.Status = switchDead
}
if h, ok := incidents[hb.groupKey()]; ok {
t := time.Unix(h.triggeredAt, 0).UTC()
src.LastTriggeredAt = &t
st.LastTriggeredAt = later(st.LastTriggeredAt, h.triggeredAt)
if h.openID != 0 {
id := h.openID
src.IncidentID = &id
if st.OpenIncidentID == nil || id > *st.OpenIncidentID {
st.OpenIncidentID = &id
}
}
}
st.LastHeartbeatAt = later(st.LastHeartbeatAt, hb.receivedAt)
st.Sources = append(st.Sources, src)
switch {
case src.Status == switchDead:
st.Status = switchDead
case st.Status == switchDormant:
st.Status = switchHealthy
}
}
// Dead ones first, then by fingerprint: what needs attention leads, and
// the order does not shuffle between refreshes.
sort.Slice(st.Sources, func(i, j int) bool {
a, b := st.Sources[i], st.Sources[j]
if (a.Status == switchDead) != (b.Status == switchDead) {
return a.Status == switchDead
}
return a.Fingerprint < b.Fingerprint
})
out = append(out, st)
}
return out, nil
}
+164 -14
View File
@@ -1,6 +1,7 @@
package api_test
import (
"context"
"net/http"
"strings"
"testing"
@@ -483,13 +484,13 @@ func TestDeadman_ConfigurationIsPerTeam(t *testing.T) {
unwatched := newTeam(t, s, "unwatched")
// Only the first team calls Watchdog a heartbeat.
resp := s.req(t, http.MethodPut, "/api/teams/"+id64(watched.id)+"/deadman", map[string]any{
"matchers": "alertname=Watchdog",
resp := s.req(t, http.MethodPost, "/api/teams/"+id64(watched.id)+"/deadman/switches", map[string]any{
"matcher": "alertname=Watchdog",
"timeout_seconds": 3600,
"severity": "critical",
})
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
if resp.StatusCode != http.StatusCreated {
t.Fatalf("configure the watched team: %d", resp.StatusCode)
}
@@ -550,9 +551,9 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
decode(t, s.req(t, http.MethodPost, "/api/users/"+id64(user.ID)+"/api-keys",
map[string]string{"name": "test"}), &key)
req, _ := http.NewRequest(http.MethodPut,
s.URL+"/api/teams/"+id64(team.id)+"/deadman",
strings.NewReader(`{"matchers":"alertname=Watchdog","timeout_seconds":60}`))
req, _ := http.NewRequest(http.MethodPost,
s.URL+"/api/teams/"+id64(team.id)+"/deadman/switches",
strings.NewReader(`{"matcher":"alertname=Watchdog","timeout_seconds":60}`))
req.Header.Set("Authorization", "Bearer "+key.Key)
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
@@ -564,7 +565,7 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
t.Errorf("a member editing the switches: expected 403, got %d", resp.StatusCode)
}
read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman", nil)
read, _ := http.NewRequest(http.MethodGet, s.URL+"/api/teams/"+id64(team.id)+"/deadman/switches", nil)
read.Header.Set("Authorization", "Bearer "+key.Key)
got, err := http.DefaultClient.Do(read)
if err != nil {
@@ -577,16 +578,165 @@ func TestDeadman_ConfigurationIsOwnerOnly(t *testing.T) {
}
// A matcher with no alertname watches nothing, silently, which is the failure
// this feature exists to prevent — so it is refused at the door.
func TestDeadman_UnusableMatchersAreRejected(t *testing.T) {
// this feature exists to prevent — so it is refused at the door, along with the
// other things that would make a switch unable to fire.
func TestDeadman_UnusableSwitchesAreRejected(t *testing.T) {
s, _ := deadmanTS(t, deadmanCfg())
resp := s.req(t, http.MethodPut, "/api/teams/"+defaultTeam+"/deadman", map[string]any{
"matchers": "cluster=prod",
"timeout_seconds": 900,
})
for name, body := range map[string]map[string]any{
"no alertname": {"matcher": "cluster=prod", "timeout_seconds": 900},
"malformed": {"matcher": "alertname=Watchdog,garbage", "timeout_seconds": 900},
"several": {"matcher": "alertname=A; alertname=B", "timeout_seconds": 900},
"zero timeout": {"matcher": "alertname=Watchdog", "timeout_seconds": 0},
"bad severity": {"matcher": "alertname=Watchdog", "timeout_seconds": 900, "severity": "loud"},
"empty matcher": {"matcher": "", "timeout_seconds": 900},
} {
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", body)
resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {
t.Errorf("expected 400 for a matcher with no alertname, got %d", resp.StatusCode)
t.Errorf("%s: expected 400, got %d", name, resp.StatusCode)
}
}
}
// ---------------------------------------------------------------------------
// The switch list
// ---------------------------------------------------------------------------
// listSwitches reads the default team's switches as the Switches page does.
func listSwitches(t *testing.T, s *ts) []map[string]any {
t.Helper()
return list(t, s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/deadman/switches", nil))
}
// A switch is healthy while its heartbeat is fresh, dead once it is silent, and
// dormant until the first one arrives.
func TestDeadman_ListReportsStatus(t *testing.T) {
s, _ := deadmanTS(t, api.ParseDeadmanConfig("alertname=Watchdog; alertname=NeverSent", time.Hour, "critical"))
got := listSwitches(t, s)
if len(got) != 2 {
t.Fatalf("expected 2 switches, got %d", len(got))
}
for _, sw := range got {
if sw["status"] != "dormant" || sw["last_heartbeat_at"] != nil || sw["last_triggered_at"] != nil {
t.Errorf("a switch nobody has heard from should be dormant and blank, got %v", sw)
}
}
heartbeat(t, s, "fp-watchdog", nil)
got = listSwitches(t, s)
if got[0]["status"] != "healthy" || got[0]["last_heartbeat_at"] == nil {
t.Errorf("a fresh heartbeat should be healthy with a timestamp, got %v", got[0])
}
if got[1]["status"] != "dormant" {
t.Errorf("the other switch is still dormant, got %v", got[1]["status"])
}
silence(t, s, "fp-watchdog", 2*time.Hour)
sweep(t, s, noArchive)
got = listSwitches(t, s)
if got[0]["status"] != "dead" {
t.Fatalf("a silent heartbeat should be dead, got %v", got[0]["status"])
}
if got[0]["last_triggered_at"] == nil || got[0]["open_incident_id"] == nil {
t.Errorf("a dead switch should show when it triggered and its open incident, got %v", got[0])
}
}
// One matcher, several clusters: the switch is as bad as its worst heartbeat and
// each heartbeat is listed on its own.
func TestDeadman_ListBreaksDownByFingerprint(t *testing.T) {
s, _ := deadmanTS(t, deadmanCfg())
heartbeat(t, s, "fp-a", map[string]string{"cluster": "a"})
heartbeat(t, s, "fp-b", map[string]string{"cluster": "b"})
silence(t, s, "fp-b", 2*time.Hour)
sw := listSwitches(t, s)[0]
if sw["status"] != "dead" {
t.Errorf("one dead cluster makes the switch dead, got %v", sw["status"])
}
sources := sw["sources"].([]any)
if len(sources) != 2 {
t.Fatalf("expected 2 sources, got %d", len(sources))
}
first, second := sources[0].(map[string]any), sources[1].(map[string]any)
if first["fingerprint"] != "fp-b" || first["status"] != "dead" || second["status"] != "healthy" {
t.Errorf("the dead source should lead, got %v then %v", first, second)
}
}
// Every switch keeps its own deadline.
func TestDeadman_TimeoutsArePerSwitch(t *testing.T) {
s, _ := deadmanTS(t, deadmanCfg())
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/deadman/switches", map[string]any{
"matcher": "alertname=Edge", "timeout_seconds": 300,
})
resp.Body.Close()
heartbeat(t, s, "fp-watchdog", nil)
postWebhook(t, s, []map[string]any{
amAlert("fp-edge", "Edge", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
}, `{}:{alertname="Edge"}`)
// Ten minutes of silence: past the Edge switch's five, inside Watchdog's hour.
silence(t, s, "fp-watchdog", 10*time.Minute)
silence(t, s, "fp-edge", 10*time.Minute)
got := listSwitches(t, s)
if got[0]["status"] != "healthy" || got[1]["status"] != "dead" {
t.Errorf("want Watchdog healthy and Edge dead, got %v and %v", got[0]["status"], got[1]["status"])
}
}
// Deleting is an owner's, is scoped to the team, and leaves what the switch
// already opened alone.
func TestDeadman_DeleteIsScopedToTheTeam(t *testing.T) {
s, _ := deadmanTS(t, deadmanCfg())
other := newTeam(t, s, "other")
id := int64(listSwitches(t, s)[0]["id"].(float64))
// Another team's owner cannot reach it.
resp := other.call(http.MethodDelete, "/api/teams/"+id64(other.id)+"/deadman/switches/"+id64(id), nil)
resp.Body.Close()
if resp.StatusCode != http.StatusNotFound {
t.Errorf("deleting another team's switch: expected 404, got %d", resp.StatusCode)
}
if got := len(listSwitches(t, s)); got != 1 {
t.Fatalf("the switch should have survived, %d left", got)
}
resp = s.req(t, http.MethodDelete, "/api/teams/"+defaultTeam+"/deadman/switches/"+id64(id), nil)
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("deleting: expected 204, got %d", resp.StatusCode)
}
if got := len(listSwitches(t, s)); got != 0 {
t.Errorf("expected no switches, got %d", got)
}
}
// The environment's defaults are handed out once and then belong to the teams.
func TestDeadman_SeedRunsOnce(t *testing.T) {
s := newTS(t)
cfg := api.ParseDeadmanConfig("alertname=Watchdog", time.Hour, "critical")
if err := api.SeedDeadmanConfigs(context.Background(), s.db, cfg); err != nil {
t.Fatalf("seed: %v", err)
}
if got := len(listSwitches(t, s)); got != 1 {
t.Fatalf("the first seed should add the default, got %d switches", got)
}
// The owner deletes it; a restart must not put it back.
id := int64(listSwitches(t, s)[0]["id"].(float64))
s.req(t, http.MethodDelete, "/api/teams/"+defaultTeam+"/deadman/switches/"+id64(id), nil).Body.Close()
if err := api.SeedDeadmanConfigs(context.Background(), s.db, cfg); err != nil {
t.Fatalf("seed again: %v", err)
}
if got := len(listSwitches(t, s)); got != 0 {
t.Errorf("a second seed resurrected %d switch(es)", got)
}
}
+185 -1
View File
@@ -346,8 +346,192 @@ func handleGetEscalation(db *sql.DB) http.HandlerFunc {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, escalationResponse(policy, teamID))
view, err := escalationStatus(r.Context(), db, teamID, policy)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, view)
}
}
// Level statuses, as the Escalation page colours them.
const (
levelReady = "ready"
levelEscalating = "escalating"
levelUnreachable = "unreachable"
)
// escalationTargetView is a target with who it means today and whether that
// person can actually be woken. The extra fields are output only: the PUT body
// is the plain escalationTargetJSON, and anything else in it is ignored.
type escalationTargetView struct {
escalationTargetJSON
// Username is who the target resolves to right now: the named person, or
// whoever the rota says is on call today. Empty when nobody is.
Username string `json:"username,omitempty"`
// Reachable is whether a page to this target would go anywhere, and Problem
// says why not when it would not — the same conditions pageLevel skips on.
Reachable bool `json:"reachable"`
Problem string `json:"problem,omitempty"`
}
type escalationLevelView struct {
Position int64 `json:"position"`
TimeoutSeconds int64 `json:"timeout_seconds"`
Targets []escalationTargetView `json:"targets"`
// Status is unreachable when no target of the level could be woken — a rung
// that looks configured and pages nobody, which is worth seeing before an
// incident finds it — escalating when an unanswered incident has climbed to
// it, and ready otherwise.
Status string `json:"status"`
// Waiting lists the open, unacknowledged incidents currently on this level.
Waiting []int64 `json:"waiting"`
}
type escalationView struct {
TeamID int64 `json:"team_id"`
RepeatCount int64 `json:"repeat_count"`
FallbackTopic string `json:"fallback_topic"`
Levels []escalationLevelView `json:"levels"`
// LastEscalatedAt is when an incident of this team last moved up the ladder,
// or ran off the end of it, and LastEscalatedIncidentID which one. Absent
// when nothing ever has: a ladder nobody has needed yet.
LastEscalatedAt *time.Time `json:"last_escalated_at,omitempty"`
LastEscalatedIncidentID *int64 `json:"last_escalated_incident_id,omitempty"`
}
// escalationStatus is a team's ladder together with what it would do right now
// and what it has been doing. The resolution follows pageLevel's rules, so the
// page cannot promise a page that the notifier would skip.
func escalationStatus(ctx context.Context, db *sql.DB, teamID int64, policy *escalationPolicy) (escalationView, error) {
base := escalationResponse(policy, teamID)
out := escalationView{
TeamID: teamID, RepeatCount: base.RepeatCount, FallbackTopic: base.FallbackTopic,
Levels: []escalationLevelView{},
}
if !policy.configured() {
return out, nil
}
onCall, err := currentOnCall(ctx, db, teamID)
if err != nil {
return out, err
}
type account struct {
username string
topic bool
disabled bool
}
accounts := map[int64]account{}
lookup := func(id int64) (account, error) {
if a, ok := accounts[id]; ok {
return a, nil
}
var a account
var topic *string
var disabledAt *int64
if err := db.QueryRowContext(ctx,
"SELECT username, ntfy_topic, disabled_at FROM users WHERE id = $1", id).
Scan(&a.username, &topic, &disabledAt); err != nil {
return a, err
}
a.topic = topic != nil && *topic != ""
a.disabled = disabledAt != nil
accounts[id] = a
return a, nil
}
waiting := map[int64][]int64{}
rows, err := db.QueryContext(ctx, `
SELECT id, escalation_level FROM incidents
WHERE team_id = $1 AND resolved_at IS NULL AND archived_at IS NULL
AND status = 'triggered' AND escalation_level > 0
ORDER BY id`, teamID)
if err != nil {
return out, err
}
for rows.Next() {
var id, level int64
if err := rows.Scan(&id, &level); err != nil {
rows.Close()
return out, err
}
waiting[level] = append(waiting[level], id)
}
rows.Close()
if err := rows.Err(); err != nil {
return out, err
}
for _, l := range base.Levels {
level := escalationLevelView{
Position: l.Position, TimeoutSeconds: l.TimeoutSeconds,
Targets: []escalationTargetView{}, Waiting: []int64{},
}
if w := waiting[l.Position]; w != nil {
level.Waiting = w
}
anyReachable := false
for _, t := range l.Targets {
view := escalationTargetView{escalationTargetJSON: t}
userID := t.UserID
if t.Kind == "oncall" {
userID = onCall
}
switch {
case userID == nil:
view.Problem = "nobody is on call today"
default:
a, err := lookup(*userID)
switch {
case err != nil:
view.Problem = "account not found"
case a.disabled:
view.Username, view.Problem = a.username, "account is disabled"
case !a.topic:
view.Username, view.Problem = a.username, "has no ntfy topic"
default:
view.Username, view.Reachable = a.username, true
}
}
anyReachable = anyReachable || view.Reachable
level.Targets = append(level.Targets, view)
}
switch {
case !anyReachable:
level.Status = levelUnreachable
case l.Position >= 2 && len(level.Waiting) > 0:
level.Status = levelEscalating
default:
level.Status = levelReady
}
out.Levels = append(out.Levels, level)
}
var incidentID, at int64
switch err := db.QueryRowContext(ctx, `
SELECT e.incident_id, e.created_at
FROM incident_events e JOIN incidents i ON i.id = e.incident_id
WHERE i.team_id = $1 AND e.type = $2
ORDER BY e.created_at DESC, e.id DESC LIMIT 1`, teamID, evEscalated).
Scan(&incidentID, &at); {
case err == sql.ErrNoRows:
case err != nil:
return out, err
default:
t := time.Unix(at, 0).UTC()
out.LastEscalatedAt, out.LastEscalatedIncidentID = &t, &incidentID
}
return out, nil
}
type escalationLevelJSON struct {
+119
View File
@@ -383,3 +383,122 @@ func TestEscalation_SkipsUnreachableTargets(t *testing.T) {
t.Errorf("a target with no topic should page nothing, paged %v", got)
}
}
// ---------------------------------------------------------------------------
// The ladder as the Escalation page reads it
// ---------------------------------------------------------------------------
type ladderLevel struct {
Status string `json:"status"`
Waiting []int64 `json:"waiting"`
Targets []struct {
Kind string `json:"kind"`
Username string `json:"username"`
Reachable bool `json:"reachable"`
Problem string `json:"problem"`
} `json:"targets"`
}
type ladderView struct {
Levels []ladderLevel `json:"levels"`
LastEscalatedAt *string `json:"last_escalated_at"`
LastEscalatedIncidentID *int64 `json:"last_escalated_incident_id"`
}
func readLadder(t *testing.T, s *ts) ladderView {
t.Helper()
var v ladderView
decode(t, s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/escalation", nil), &v)
return v
}
// Targets say who they mean today, so "whoever is on call" is a name and not a
// promise.
func TestEscalation_StatusResolvesTargets(t *testing.T) {
s, _ := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
second := teamUser(t, s, "second", "terdut-second")
ladder(t, s, second, 0, "terdut-fallback")
v := readLadder(t, s)
if len(v.Levels) != 2 {
t.Fatalf("expected 2 levels, got %d", len(v.Levels))
}
if got := v.Levels[0].Targets[0]; got.Kind != "oncall" || got.Username != "admin" || !got.Reachable {
t.Errorf("the rota target should resolve to the person on call, got %+v", got)
}
if got := v.Levels[1].Targets[0]; got.Username != "second" || !got.Reachable {
t.Errorf("the named target should be reachable, got %+v", got)
}
if v.Levels[0].Status != "ready" || v.Levels[1].Status != "ready" || v.LastEscalatedAt != nil {
t.Errorf("an idle, healthy ladder is ready and has never escalated, got %+v", v)
}
}
// A rung that would page nobody is called out before an incident finds it.
func TestEscalation_StatusFlagsUnreachableLevels(t *testing.T) {
s, _ := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
silent := teamUser(t, s, "silent", "terdut-silent")
ladder(t, s, silent, 0, "terdut-fallback")
// Nobody on call today, and the named person loses their topic.
s.exec(t, "DELETE FROM schedule_entries")
s.exec(t, "UPDATE users SET ntfy_topic = NULL WHERE id = $1", silent)
v := readLadder(t, s)
if v.Levels[0].Status != "unreachable" || v.Levels[0].Targets[0].Problem != "nobody is on call today" {
t.Errorf("an empty rota should make level 1 unreachable, got %+v", v.Levels[0])
}
if v.Levels[1].Status != "unreachable" || v.Levels[1].Targets[0].Problem != "has no ntfy topic" {
t.Errorf("a person with no topic should make level 2 unreachable, got %+v", v.Levels[1])
}
s.exec(t, "UPDATE users SET disabled_at = 1 WHERE id = $1", silent)
if p := readLadder(t, s).Levels[1].Targets[0].Problem; p != "account is disabled" {
t.Errorf("a disabled account should say so, got %q", p)
}
}
// Where unanswered incidents are right now, and when the ladder last did its
// job.
func TestEscalation_StatusShowsWhoIsWaitingAndLastEscalation(t *testing.T) {
s, _ := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
second := teamUser(t, s, "second", "terdut-second")
ladder(t, s, second, 0, "terdut-fallback")
postWebhook(t, s, []map[string]any{
amAlert("fp-wait", "DiskFull", "firing", "2026-05-20T10:00:00Z", zeroTime, nil),
})
s.sweepNotify(t)
// On level 1 it is waiting, which is normal and not yet an escalation.
v := readLadder(t, s)
if len(v.Levels[0].Waiting) != 1 || v.Levels[0].Status != "ready" || v.LastEscalatedAt != nil {
t.Fatalf("a fresh incident waits on level 1 quietly, got %+v", v)
}
overdue(t, s, 1)
s.sweepNotify(t)
v = readLadder(t, s)
if v.Levels[1].Status != "escalating" || len(v.Levels[1].Waiting) != 1 || v.Levels[1].Waiting[0] != 1 {
t.Errorf("level 2 should be escalating with the incident on it, got %+v", v.Levels[1])
}
if v.LastEscalatedAt == nil || v.LastEscalatedIncidentID == nil || *v.LastEscalatedIncidentID != 1 {
t.Errorf("the escalation should be recorded, got %+v", v)
}
// Somebody answers: nothing is waiting, but the history stays.
s.req(t, http.MethodPost, "/api/incidents/1/acknowledge", nil).Body.Close()
v = readLadder(t, s)
if v.Levels[1].Status != "ready" || len(v.Levels[1].Waiting) != 0 || v.LastEscalatedAt == nil {
t.Errorf("an acknowledged incident stops waiting but stays in the history, got %+v", v)
}
}
// No ladder is a real answer, not an error.
func TestEscalation_StatusWithoutALadder(t *testing.T) {
s, _ := notifyTS(t, api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute})
v := readLadder(t, s)
if len(v.Levels) != 0 || v.LastEscalatedAt != nil {
t.Errorf("a team with no ladder should read as empty, got %+v", v)
}
}
+32
View File
@@ -36,6 +36,10 @@ const (
evUnsnoozed = "unsnoozed"
evResolved = "resolved"
evNote = "note"
// evResolutionNote is the note worth finding again: what fixed it. The
// similar-incidents lookup and the page lead with these; plain notes are
// the working chatter and stay one click away.
evResolutionNote = "resolution_note"
evDeadmanSilent = "deadman_silent"
)
@@ -295,6 +299,34 @@ func openIncidentForAlert(ctx context.Context, q querier, alertID int64) (int64,
return id, err
}
// volatileLabels say where a problem ran this time, not what the problem is, so
// they stay out of the signature. Migration 008's backfill lists the same set.
var volatileLabels = map[string]bool{
"instance": true, "pod": true, "pod_name": true, "pod_ip": true,
"container": true, "container_name": true, "endpoint": true,
}
// incidentSignature identifies "the same problem" across incidents: the alert
// name plus the stable group labels, sorted. Incidents in one team with equal
// signatures are what the similar-incidents lookup returns. title stands in for
// the name when the payload carried no alertname (groupless and dead man's
// switch incidents).
func incidentSignature(groupLabels map[string]string, title string) string {
name := groupLabels["alertname"]
if name == "" {
name = title
}
rest := make([]string, 0, len(groupLabels))
for k, v := range groupLabels {
if k == "alertname" || volatileLabels[k] {
continue
}
rest = append(rest, k+"="+v)
}
sort.Strings(rest)
return name + "|" + strings.Join(rest, ",")
}
// incidentTitle renders a human-readable title from Alertmanager's groupLabels,
// leading with the alert name and appending whatever else the operator grouped
// by. Falls back to the alert's own name when the payload carried no groupLabels.
+26 -4
View File
@@ -3,6 +3,7 @@ package api
import (
"database/sql"
"fmt"
"io"
"net/http"
"strconv"
"strings"
@@ -237,6 +238,15 @@ func handleIncidentResolve(db *sql.DB) http.HandlerFunc {
return
}
user, _ := userFromContext(r.Context())
// The body is optional: clients that predate resolution notes send none.
var req struct {
Resolution string `json:"resolution"`
}
if err := decodeJSON(r, &req); err != nil && err != io.EOF {
respond(w, http.StatusBadRequest, errResp("invalid request body"))
return
}
req.Resolution = strings.TrimSpace(req.Resolution)
if !updateOpenIncident(w, r, db, id,
`UPDATE incidents SET status = 'resolved', resolved_at = $1, resolution_source = $2
WHERE id = $3 AND resolved_at IS NULL`,
@@ -252,6 +262,12 @@ func handleIncidentResolve(db *sql.DB) http.HandlerFunc {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if req.Resolution != "" {
if err := logEvent(r.Context(), db, id, evResolutionNote, &user.ID, nil, &req.Resolution); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
}
respondIncident(w, r, db, id)
}
}
@@ -421,11 +437,17 @@ func handleCreateNote(db *sql.DB) http.HandlerFunc {
}
var req struct {
Content string `json:"content"`
// Pinned files the note as the resolution note: what fixed it.
Pinned bool `json:"pinned"`
}
if err := decodeJSON(r, &req); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid request body"))
return
}
noteType := evNote
if req.Pinned {
noteType = evResolutionNote
}
if req.Content == "" {
respond(w, http.StatusBadRequest, errResp("content is required"))
return
@@ -440,7 +462,7 @@ func handleCreateNote(db *sql.DB) http.HandlerFunc {
err := db.QueryRowContext(r.Context(), `
INSERT INTO incident_events (incident_id, type, user_id, detail, created_at)
VALUES ($1, $2, $3, $4, $5)
RETURNING id`, id, evNote, user.ID, req.Content, now.Unix()).Scan(&eventID)
RETURNING id`, id, noteType, user.ID, req.Content, now.Unix()).Scan(&eventID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
@@ -449,7 +471,7 @@ func handleCreateNote(db *sql.DB) http.HandlerFunc {
respond(w, http.StatusCreated, models.IncidentEvent{
ID: eventID,
IncidentID: id,
Type: evNote,
Type: noteType,
UserID: &user.ID,
Username: &user.Username,
Detail: &req.Content,
@@ -475,8 +497,8 @@ func handleDeleteNote(db *sql.DB) http.HandlerFunc {
user, _ := userFromContext(r.Context())
res, err := db.ExecContext(r.Context(), `
DELETE FROM incident_events
WHERE id = $1 AND incident_id = $2 AND type = $3 AND user_id = $4`,
eventID, id, evNote, user.ID)
WHERE id = $1 AND incident_id = $2 AND type IN ($3, $4) AND user_id = $5`,
eventID, id, evNote, evResolutionNote, user.ID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
+154
View File
@@ -0,0 +1,154 @@
package api_test
import (
"net/http"
"testing"
"time"
"git.ryuvia.com/niklas/terdut-server/internal/api"
)
func testNotify() api.NotifyConfig {
return api.NotifyConfig{PublicURL: "https://terdut.example.com", RepeatEvery: 15 * time.Minute}
}
type memberView struct {
Username string `json:"username"`
Role string `json:"role"`
Status string `json:"status"`
OnCall bool `json:"on_call"`
NextShift *string `json:"next_shift"`
Pageable bool `json:"pageable"`
Problem string `json:"problem"`
LastActiveAt *string `json:"last_active_at"`
}
func readMembers(t *testing.T, s *ts) map[string]memberView {
t.Helper()
var list []memberView
decode(t, s.req(t, http.MethodGet, "/api/teams/"+defaultTeam+"/members", nil), &list)
out := map[string]memberView{}
for _, m := range list {
out[m.Username] = m
}
return out
}
// The list says who is on call, who could not be woken, and who is merely
// there — and an on-call person who cannot be paged is the red one.
func TestMembers_StatusReflectsRotaAndPageability(t *testing.T) {
s, _ := notifyTS(t, testNotify()) // admin is on call today, with a topic
teamUser(t, s, "reachable", "terdut-reachable")
silent := teamUser(t, s, "silent", "terdut-silent")
s.exec(t, "UPDATE users SET ntfy_topic = NULL WHERE id = $1", silent)
got := readMembers(t, s)
if m := got["admin"]; m.Status != "oncall" || !m.OnCall || !m.Pageable {
t.Errorf("the person on call should read on call, got %+v", m)
}
if m := got["reachable"]; m.Status != "reachable" || m.OnCall {
t.Errorf("a member with a topic who is off the rota is reachable, got %+v", m)
}
if m := got["silent"]; m.Status != "unpageable" || m.Problem != "has no ntfy topic" {
t.Errorf("no topic means they cannot be paged, got %+v", m)
}
// Being on call does not rescue an account that cannot be woken.
s.exec(t, "UPDATE users SET ntfy_topic = NULL WHERE username = 'admin'")
if m := readMembers(t, s)["admin"]; m.Status != "unpageable" || !m.OnCall {
t.Errorf("an on-call person with no topic is the red case, got %+v", m)
}
s.exec(t, "UPDATE users SET disabled_at = 1 WHERE id = $1", silent)
if m := readMembers(t, s)["silent"]; m.Problem != "account is disabled" {
t.Errorf("a disabled account should say so, got %+v", m)
}
}
// The next shift is the next day after today, not today itself.
func TestMembers_NextShiftIsAfterToday(t *testing.T) {
s, _ := notifyTS(t, testNotify())
tomorrow := time.Now().UTC().AddDate(0, 0, 3).Format("2006-01-02")
resp := s.req(t, http.MethodPost, "/api/teams/"+defaultTeam+"/schedule",
map[string]any{"user_id": 1, "dates": []string{tomorrow}})
resp.Body.Close()
m := readMembers(t, s)["admin"]
if !m.OnCall || m.NextShift == nil || *m.NextShift != tomorrow {
t.Errorf("want on call today with the next shift on %s, got %+v", tomorrow, m)
}
teamUser(t, s, "idle", "terdut-idle")
if m := readMembers(t, s)["idle"]; m.NextShift != nil {
t.Errorf("somebody not on the rota has no next shift, got %v", *m.NextShift)
}
}
// Last active is the newer of a session and an API key, and absent when neither
// has ever been used.
func TestMembers_LastActive(t *testing.T) {
s, _ := notifyTS(t, testNotify())
idle := teamUser(t, s, "idle", "terdut-idle")
if m := readMembers(t, s)["idle"]; m.LastActiveAt != nil {
t.Errorf("nobody has used idle's account, got %v", *m.LastActiveAt)
}
old := time.Now().Add(-48 * time.Hour).Unix()
s.exec(t, `INSERT INTO api_keys (user_id, key_hash, name, last_used_at) VALUES ($1, 'h1', 'k', $2)`, idle, old)
s.exec(t, `INSERT INTO sessions (token_hash, user_id, created_at, last_seen_at, expires_at)
VALUES ('h2', $1, $2, $3, $4)`, idle, old, old+3600, time.Now().Add(time.Hour).Unix())
m := readMembers(t, s)["idle"]
if m.LastActiveAt == nil {
t.Fatal("expected a last active time")
}
got, _ := time.Parse(time.RFC3339, *m.LastActiveAt)
if got.Unix() != old+3600 {
t.Errorf("last active should be the newer session (%d), got %d", old+3600, got.Unix())
}
}
// The last owner can be neither removed nor demoted; with another owner in
// place, both are fine.
func TestMembers_LastOwnerIsProtected(t *testing.T) {
s, _ := notifyTS(t, testNotify())
tm := newTeam(t, s, "red")
base := "/api/teams/" + id64(tm.id) + "/members"
// Creating a team makes the creator an owner too; step the admin out so
// "red-user" is the only one left.
resp := s.req(t, http.MethodDelete, base+"/1", nil)
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("removing the creator: %d", resp.StatusCode)
}
var members []map[string]any
decode(t, tm.call(http.MethodGet, base, nil), &members)
var owner int64
for _, m := range members {
if m["username"] == "red-user" {
owner = int64(m["user_id"].(float64))
}
}
resp = tm.call(http.MethodPost, base, map[string]any{"user_id": owner, "role": "member"})
resp.Body.Close()
if resp.StatusCode != http.StatusConflict {
t.Errorf("demoting the last owner: expected 409, got %d", resp.StatusCode)
}
resp = tm.call(http.MethodDelete, base+"/"+id64(owner), nil)
resp.Body.Close()
if resp.StatusCode != http.StatusConflict {
t.Errorf("removing the last owner: expected 409, got %d", resp.StatusCode)
}
// A second owner frees the first to step down.
resp = s.req(t, http.MethodPost, base, map[string]any{"user_id": 1, "role": "owner"})
resp.Body.Close()
resp = tm.call(http.MethodPost, base, map[string]any{"user_id": owner, "role": "member"})
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Errorf("demoting one of two owners: expected 204, got %d", resp.StatusCode)
}
}
+21
View File
@@ -330,6 +330,16 @@ func deliver(ctx context.Context, db *sql.DB, cfg NotifyConfig, n outboxRow) err
msg := renderNotification(inc, n, firing, cfg)
// The page that opens an incident carries what fixed it last time, so the
// person woken up starts from that. Best effort: a failed lookup must not
// hold back the page itself.
if n.kind == notifyTriggered {
if sim, err := similarIncidents(ctx, db, n.incidentID, 1); err == nil && len(sim) > 0 && len(sim[0].ResolutionNotes) > 0 {
notes := sim[0].ResolutionNotes
msg.Message += "\nLast time: " + shorten(derefString(notes[len(notes)-1].Detail), 160)
}
}
// An Acknowledge button needs both a user to attribute the acknowledgement
// to and a URL the phone can reach. Minted per delivery, so every push
// carries its own short-lived token rather than reusing one.
@@ -573,6 +583,17 @@ func plural(n int) string {
return "s"
}
// shorten cuts s to at most n runes, marking the cut, and flattens newlines so
// a multi-line note stays one line in a push.
func shorten(s string, n int) string {
s = strings.Join(strings.Fields(s), " ")
r := []rune(s)
if len(r) <= n {
return s
}
return string(r[:n-1]) + "…"
}
// derefString reads a nullable text column as a plain string.
func derefString(s *string) string {
if s == nil {
+5 -2
View File
@@ -112,6 +112,7 @@ func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config) http.Handler
r.Get("/api/incidents/{id}", handleGetIncident(db))
r.Get("/api/incidents/{id}/alerts", handleIncidentAlerts(db))
r.Get("/api/incidents/{id}/timeline", handleIncidentTimeline(db))
r.Get("/api/incidents/{id}/similar", handleIncidentSimilar(db))
r.Post("/api/incidents/{id}/acknowledge", handleIncidentAcknowledge(db))
r.Delete("/api/incidents/{id}/acknowledge", handleIncidentUnacknowledge(db))
r.Post("/api/incidents/{id}/resolve", handleIncidentResolve(db))
@@ -143,12 +144,14 @@ func NewRouter(db *sql.DB, notify NotifyConfig, cfg config.Config) http.Handler
// A team's own dead man's switches: which of its alerts are heartbeats,
// and how long a silence has to last before somebody is paged.
r.Get("/api/teams/{teamID}/deadman", handleGetTeamDeadman(db))
r.Put("/api/teams/{teamID}/deadman", handleSetTeamDeadman(db))
r.Get("/api/teams/{teamID}/deadman/switches", handleListTeamDeadman(db))
r.Post("/api/teams/{teamID}/deadman/switches", handleCreateTeamDeadman(db))
r.Delete("/api/teams/{teamID}/deadman/switches/{switchID}", handleDeleteTeamDeadman(db))
// Integrations: where a team's alerts come in, and the key that says so.
r.Get("/api/teams/{teamID}/integrations", handleListIntegrations(db))
r.Post("/api/teams/{teamID}/integrations", handleCreateIntegration(db, notify.PublicURL))
r.Patch("/api/teams/{teamID}/integrations/{integrationID}", handleRenameIntegration(db))
r.Delete("/api/teams/{teamID}/integrations/{integrationID}", handleDeleteIntegration(db))
// The rota is per team. /api/schedule/current is the exception: it
+113
View File
@@ -0,0 +1,113 @@
package api
import (
"context"
"database/sql"
"net/http"
"strconv"
"time"
"git.ryuvia.com/niklas/terdut-server/internal/models"
)
const (
similarDefaultLimit = 5
similarMaxLimit = 20
)
// handleIncidentSimilar lists earlier, resolved incidents in the same team with
// the same signature that someone left notes on, incidents with a resolution
// note first. This is the "have we seen this before" answer for a responder
// looking at a fresh incident; the plain notes are one timeline fetch away.
func handleIncidentSimilar(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id, ok := incidentIDParam(w, r, db)
if !ok {
return
}
limit := similarDefaultLimit
if v := r.URL.Query().Get("limit"); v != "" {
n, err := strconv.Atoi(v)
if err != nil || n < 1 {
respond(w, http.StatusBadRequest, errResp("invalid limit"))
return
}
limit = min(n, similarMaxLimit)
}
out, err := similarIncidents(r.Context(), db, id, limit)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, out)
}
}
func similarIncidents(ctx context.Context, q querier, id int64, limit int) ([]models.SimilarIncident, error) {
rows, err := q.QueryContext(ctx, `
SELECT o.id, o.title, o.triggered_at, o.resolved_at,
(SELECT COUNT(*) FROM incident_events e
WHERE e.incident_id = o.id AND e.type = $3)
FROM incidents i
JOIN incidents o ON o.team_id = i.team_id AND o.signature = i.signature
WHERE i.id = $1 AND o.id <> i.id AND o.resolved_at IS NOT NULL
AND EXISTS (SELECT 1 FROM incident_events e
WHERE e.incident_id = o.id AND e.type IN ($3, $4))
ORDER BY EXISTS (SELECT 1 FROM incident_events e
WHERE e.incident_id = o.id AND e.type = $4) DESC,
o.triggered_at DESC
LIMIT $2`, id, limit, evNote, evResolutionNote)
if err != nil {
return nil, err
}
defer rows.Close()
out := []models.SimilarIncident{}
ids := []int64{}
for rows.Next() {
var s models.SimilarIncident
var triggered, resolved int64
if err := rows.Scan(&s.ID, &s.Title, &triggered, &resolved, &s.NoteCount); err != nil {
return nil, err
}
s.TriggeredAt = time.Unix(triggered, 0).UTC()
s.ResolvedAt = time.Unix(resolved, 0).UTC()
s.ResolutionNotes = []models.IncidentEvent{}
out = append(out, s)
ids = append(ids, s.ID)
}
if err := rows.Err(); err != nil {
return nil, err
}
if len(out) == 0 {
return out, nil
}
nrows, err := q.QueryContext(ctx, `
SELECT e.id, e.incident_id, e.type, e.user_id, u.username, e.detail, e.created_at
FROM incident_events e
LEFT JOIN users u ON u.id = e.user_id
WHERE e.incident_id = ANY($1) AND e.type = $2
ORDER BY e.created_at, e.id`, ids, evResolutionNote)
if err != nil {
return nil, err
}
defer nrows.Close()
byID := make(map[int64]*models.SimilarIncident, len(out))
for i := range out {
byID[out[i].ID] = &out[i]
}
for nrows.Next() {
var e models.IncidentEvent
var ts int64
if err := nrows.Scan(&e.ID, &e.IncidentID, &e.Type, &e.UserID, &e.Username, &e.Detail, &ts); err != nil {
return nil, err
}
e.CreatedAt = time.Unix(ts, 0).UTC()
s := byID[e.IncidentID]
s.ResolutionNotes = append(s.ResolutionNotes, e)
}
return out, nrows.Err()
}
+99
View File
@@ -0,0 +1,99 @@
package api_test
import (
"bytes"
"encoding/json"
"net/http"
"strconv"
"testing"
)
// postGrouped posts a firing webhook whose group labels are exactly the given
// map, unlike postWebhook, which only ever groups by alertname.
func postGrouped(t *testing.T, s *ts, fingerprint, startsAt string, groupLabels map[string]string) {
t.Helper()
labels := map[string]string{}
for k, v := range groupLabels {
labels[k] = v
}
payload := map[string]any{
"version": "4", "status": "firing",
"groupKey": fingerprint,
"groupLabels": groupLabels,
"alerts": []map[string]any{
amAlert(fingerprint, groupLabels["alertname"], "firing", startsAt, zeroTime, labels),
},
}
data, _ := json.Marshal(payload)
resp, err := http.Post(s.URL+"/api/integrations/"+s.ingestKey+"/alertmanager",
"application/json", bytes.NewReader(data))
if err != nil {
t.Fatalf("post webhook: %v", err)
}
resp.Body.Close()
}
func similar(t *testing.T, s *ts, id int) []map[string]any {
t.Helper()
var out []map[string]any
decode(t, s.req(t, http.MethodGet, "/api/incidents/"+strconv.Itoa(id)+"/similar", nil), &out)
return out
}
// Same alert on another instance is the same problem; a resolution note left on
// the first one is what the second one should be shown.
func TestSimilar_IgnoresVolatileLabelsAndLeadsWithResolutionNote(t *testing.T) {
s := newTS(t)
postGrouped(t, s, "fp-a", "2026-05-20T10:00:00Z",
map[string]string{"alertname": "DiskFull", "instance": "web-1", "job": "node"})
s.req(t, http.MethodPost, "/api/incidents/1/resolve",
map[string]string{"resolution": "rotated the logs"}).Body.Close()
postGrouped(t, s, "fp-b", "2026-05-21T10:00:00Z",
map[string]string{"alertname": "DiskFull", "instance": "web-2", "job": "node"})
got := similar(t, s, 2)
if len(got) != 1 || int(got[0]["id"].(float64)) != 1 {
t.Fatalf("expected incident 1 as the only similar one, got %v", got)
}
notes := got[0]["resolution_notes"].([]any)
if len(notes) != 1 || notes[0].(map[string]any)["detail"] != "rotated the logs" {
t.Fatalf("expected the resolution note, got %v", notes)
}
}
// A different stable label (job) is a different problem, and an incident nobody
// wrote a note on has nothing to show.
func TestSimilar_DifferentSignatureOrNoNotesIsExcluded(t *testing.T) {
s := newTS(t)
postGrouped(t, s, "fp-1", "2026-05-20T10:00:00Z",
map[string]string{"alertname": "DiskFull", "job": "node"})
s.req(t, http.MethodPost, "/api/incidents/1/notes", map[string]string{"content": "checked"}).Body.Close()
s.req(t, http.MethodPost, "/api/incidents/1/resolve", nil).Body.Close()
postGrouped(t, s, "fp-2", "2026-05-20T11:00:00Z",
map[string]string{"alertname": "DiskFull", "job": "db"})
s.req(t, http.MethodPost, "/api/incidents/2/resolve", nil).Body.Close()
postGrouped(t, s, "fp-3", "2026-05-21T10:00:00Z",
map[string]string{"alertname": "DiskFull", "job": "db"})
// Incident 3 matches 2 by signature, but 2 has no notes.
if got := similar(t, s, 3); len(got) != 0 {
t.Fatalf("expected nothing similar to incident 3, got %v", got)
}
}
// An open incident is not "earlier experience" yet, and the incident itself is
// never its own match.
func TestSimilar_OpenIncidentsAreNotListed(t *testing.T) {
s := newTS(t)
postGrouped(t, s, "fp-o1", "2026-05-20T10:00:00Z", map[string]string{"alertname": "Flap"})
s.req(t, http.MethodPost, "/api/incidents/1/notes",
map[string]any{"content": "still open", "pinned": true}).Body.Close()
postGrouped(t, s, "fp-o2", "2026-05-21T10:00:00Z", map[string]string{"alertname": "Flap"})
if got := similar(t, s, 2); len(got) != 0 {
t.Fatalf("expected an open incident not to be listed, got %v", got)
}
}
+171
View File
@@ -0,0 +1,171 @@
package api_test
import (
"bytes"
"net/http"
"strings"
"testing"
"time"
)
// listSources reads a team's alert sources as the Sources page does.
func listSources(t *testing.T, tm teamFixture) []map[string]any {
t.Helper()
return list(t, tm.call(http.MethodGet, "/api/teams/"+id64(tm.id)+"/integrations", nil))
}
// addSource mints a second source in a team and returns its key.
func addSource(t *testing.T, tm teamFixture, name string) string {
t.Helper()
var out struct {
Key string `json:"key"`
}
decode(t, tm.call(http.MethodPost, "/api/teams/"+id64(tm.id)+"/integrations",
map[string]string{"name": name}), &out)
return out.Key
}
// A source that has never posted is "never", with nothing to say about alerts.
func TestSources_NeverUsedIsBlank(t *testing.T) {
s := newTS(t)
tm := newTeam(t, s, "red")
got := listSources(t, tm)
if len(got) != 1 {
t.Fatalf("expected 1 source, got %d", len(got))
}
src := got[0]
if src["status"] != "never" || src["last_used_at"] != nil || src["last_alert_at"] != nil {
t.Errorf("a source nobody has posted on should be blank, got %v", src)
}
if src["alerts_24h"].(float64) != 0 {
t.Errorf("alerts_24h = %v, want 0", src["alerts_24h"])
}
}
// Each source is credited with what arrived on its own key, and only that.
func TestSources_AlertsAreAttributedToTheirSource(t *testing.T) {
s := newTS(t)
tm := newTeam(t, s, "red")
second := addSource(t, tm, "staging")
postToIntegration(t, s, tm.key, "fp-1", "DiskFull")
postToIntegration(t, s, tm.key, "fp-2", "CPUHot")
got := listSources(t, tm)
first, other := got[0], got[1]
if first["status"] != "active" || first["last_used_at"] == nil || first["last_alert_at"] == nil {
t.Errorf("the source that posted should be active with timestamps, got %v", first)
}
if first["alerts_24h"].(float64) != 2 {
t.Errorf("alerts_24h = %v, want 2", first["alerts_24h"])
}
if other["status"] != "never" || other["alerts_24h"].(float64) != 0 {
t.Errorf("the other source should be untouched, got %v", other)
}
// Re-sending the same alert on the other key moves it: last sender wins.
postToIntegration(t, s, second, "fp-1", "DiskFull")
got = listSources(t, tm)
if got[0]["alerts_24h"].(float64) != 1 || got[1]["alerts_24h"].(float64) != 1 {
t.Errorf("fp-1 should have moved to the second source, got %v and %v",
got[0]["alerts_24h"], got[1]["alerts_24h"])
}
}
// A payload with no alerts in it is a webhook, not an alert: the source was
// heard from, and nothing arrived.
func TestSources_EmptyPayloadStampsUseButNotAlert(t *testing.T) {
s := newTS(t)
tm := newTeam(t, s, "red")
resp, err := http.Post(s.URL+"/api/integrations/"+tm.key+"/alertmanager",
"application/json", bytes.NewReader([]byte(`{"version":"4","status":"firing","alerts":[]}`)))
if err != nil {
t.Fatalf("post: %v", err)
}
resp.Body.Close()
src := listSources(t, tm)[0]
if src["status"] != "active" || src["last_alert_at"] != nil {
t.Errorf("want active with no alert yet, got %v", src)
}
}
// Quiet is "has posted, not lately"; the alert counter forgets after a day but
// the last alert's timestamp is kept.
func TestSources_QuietAfterADay(t *testing.T) {
s := newTS(t)
tm := newTeam(t, s, "red")
postToIntegration(t, s, tm.key, "fp-1", "DiskFull")
old := time.Now().Add(-48 * time.Hour).Unix()
s.exec(t, "UPDATE integrations SET last_used_at = $1", old)
s.exec(t, "UPDATE alerts SET received_at = $1 WHERE fingerprint = 'fp-1'", old)
src := listSources(t, tm)[0]
if src["status"] != "quiet" {
t.Errorf("status = %v, want quiet", src["status"])
}
if src["alerts_24h"].(float64) != 0 {
t.Errorf("alerts_24h = %v, want 0", src["alerts_24h"])
}
if src["last_alert_at"] == nil {
t.Error("last_alert_at should survive the day")
}
}
// Revoking a source does not take its alerts with it.
func TestSources_RevokeKeepsTheAlerts(t *testing.T) {
s := newTS(t)
tm := newTeam(t, s, "red")
postToIntegration(t, s, tm.key, "fp-1", "DiskFull")
id := int64(listSources(t, tm)[0]["id"].(float64))
resp := tm.call(http.MethodDelete, "/api/teams/"+id64(tm.id)+"/integrations/"+id64(id), nil)
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("revoke: %d", resp.StatusCode)
}
if got := len(list(t, tm.call(http.MethodGet, "/api/alerts", nil))); got != 1 {
t.Errorf("the alert should outlive its source, got %d alerts", got)
}
}
// Renaming is an owner's, scoped to the team, and does not touch the key.
func TestSources_Rename(t *testing.T) {
s := newTS(t)
tm := newTeam(t, s, "red")
other := newTeam(t, s, "blue")
id := int64(listSources(t, tm)[0]["id"].(float64))
path := "/api/teams/" + id64(tm.id) + "/integrations/" + id64(id)
resp := tm.call(http.MethodPatch, path, map[string]string{"name": " prod "})
resp.Body.Close()
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("rename: %d", resp.StatusCode)
}
if name := listSources(t, tm)[0]["name"]; name != "prod" {
t.Errorf("name = %q, want it trimmed to prod", name)
}
postToIntegration(t, s, tm.key, "fp-1", "DiskFull") // the old key still works
for name, body := range map[string]map[string]string{
"empty": {"name": " "},
"too long": {"name": strings.Repeat("x", 101)},
} {
resp := tm.call(http.MethodPatch, path, body)
resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {
t.Errorf("%s name: expected 400, got %d", name, resp.StatusCode)
}
}
// Another team's owner cannot reach it.
resp = other.call(http.MethodPatch, "/api/teams/"+id64(other.id)+"/integrations/"+id64(id),
map[string]string{"name": "mine now"})
resp.Body.Close()
if resp.StatusCode != http.StatusNotFound {
t.Errorf("renaming another team's source: expected 404, got %d", resp.StatusCode)
}
}
+299 -73
View File
@@ -210,8 +210,40 @@ func handleDeleteTeam(db *sql.DB) http.HandlerFunc {
}
}
// handleListTeamMembers names everybody in a team. Visible to any member: you
// can see who else is on the rota you are on.
// Member statuses, as the Members page colours them.
const (
memberOnCall = "oncall"
memberReachable = "reachable"
memberUnpageable = "unpageable"
)
// memberStatus is a team member with what matters about them at 03:00: whether
// they are on call, whether a page to them would go anywhere, and whether they
// have been around. The extra fields are output only.
type memberStatus struct {
models.TeamMember
// Status is unpageable when a page to them would go nowhere — even when
// they are on call, since that is the case that matters most — on_call when
// the rota has them today, reachable otherwise.
Status string `json:"status"`
OnCall bool `json:"on_call"`
// NextShift is the first day after today the rota has them (YYYY-MM-DD).
NextShift *string `json:"next_shift,omitempty"`
// Pageable is whether they have an ntfy topic and an enabled account — the
// conditions pageLevel and the notifier skip on. Never the topic itself.
Pageable bool `json:"pageable"`
Problem string `json:"problem,omitempty"`
// LastActiveAt is the last time they used a session or an API key.
LastActiveAt *time.Time `json:"last_active_at,omitempty"`
}
// handleListTeamMembers names everybody in a team, with their status. Visible to
// any member: you can see who else is on the rota you are on.
func handleListTeamMembers(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
@@ -223,26 +255,56 @@ func handleListTeamMembers(db *sql.DB) http.HandlerFunc {
}
rows, err := db.QueryContext(r.Context(), `
SELECT m.team_id, m.user_id, u.username, m.role, m.joined_at
SELECT m.team_id, m.user_id, u.username, m.role, m.joined_at,
u.ntfy_topic IS NOT NULL AND u.ntfy_topic <> '',
u.disabled_at IS NOT NULL,
GREATEST(
COALESCE((SELECT MAX(last_seen_at) FROM sessions WHERE user_id = u.id), 0),
COALESCE((SELECT MAX(last_used_at) FROM api_keys WHERE user_id = u.id), 0)),
EXISTS (SELECT 1 FROM schedule_entries s
WHERE s.team_id = m.team_id AND s.user_id = u.id AND s.date = $2),
(SELECT MIN(date) FROM schedule_entries s
WHERE s.team_id = m.team_id AND s.user_id = u.id AND s.date > $2)
FROM team_members m
JOIN users u ON u.id = m.user_id
WHERE m.team_id = $1
ORDER BY u.username`, teamID)
ORDER BY u.username`, teamID, todayUTC())
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
defer rows.Close()
members := []models.TeamMember{}
members := []memberStatus{}
for rows.Next() {
var m models.TeamMember
var joined int64
if err := rows.Scan(&m.TeamID, &m.UserID, &m.Username, &m.Role, &joined); err != nil {
var m memberStatus
var joined, lastActive int64
var hasTopic, disabled bool
if err := rows.Scan(&m.TeamID, &m.UserID, &m.Username, &m.Role, &joined,
&hasTopic, &disabled, &lastActive, &m.OnCall, &m.NextShift); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
m.JoinedAt = time.Unix(joined, 0).UTC()
if lastActive > 0 {
t := time.Unix(lastActive, 0).UTC()
m.LastActiveAt = &t
}
switch {
case disabled:
m.Problem = "account is disabled"
case !hasTopic:
m.Problem = "has no ntfy topic"
}
m.Pageable = m.Problem == ""
switch {
case !m.Pageable:
m.Status = memberUnpageable
case m.OnCall:
m.Status = memberOnCall
default:
m.Status = memberReachable
}
members = append(members, m)
}
if err := rows.Err(); err != nil {
@@ -281,6 +343,20 @@ func handleAddTeamMember(db *sql.DB) http.HandlerFunc {
return
}
// Demoting the last owner is removing them by another route: the team
// would have nobody who can edit it.
if req.Role == models.RoleMember {
last, err := isLastTeamOwner(r.Context(), db, teamID, req.UserID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if last {
respond(w, http.StatusConflict, errResp("cannot demote the last owner of a team"))
return
}
}
_, err := db.ExecContext(r.Context(), `
INSERT INTO team_members (team_id, user_id, role)
VALUES ($1, $2, $3)
@@ -355,8 +431,42 @@ func isLastTeamOwner(ctx context.Context, db *sql.DB, teamID, userID int64) (boo
// Integrations
// ---------------------------------------------------------------------------
// handleListIntegrations lists a team's integrations. Never the keys: those
// exist in plaintext only in the response that created them.
// sourceQuietAfter is how long a source may go without posting before the
// Sources page calls it quiet rather than active. A day is longer than any
// repeat_interval worth having, so an Alertmanager that is up and has anything
// firing never crosses it; a source with nothing firing may, and that is a
// reason to look, not proof of a fault — which is why this is a colour and not
// an alarm. Dead man's switches are where silence pages.
const sourceQuietAfter = 24 * time.Hour
const (
sourceActive = "active"
sourceQuiet = "quiet"
sourceNever = "never"
)
// integrationStatus is an integration as the Sources page shows it.
type integrationStatus struct {
models.Integration
// Status is active when the key posted within sourceQuietAfter, quiet when
// it has posted but not lately, never when it has not posted at all.
Status string `json:"status"`
// LastAlertAt is when an alert last arrived on this source, which is not the
// same as when it last posted: a payload with nothing usable in it stamps
// last_used_at and not this. Absent until an alert has arrived since
// migration 010 started recording it.
LastAlertAt *time.Time `json:"last_alert_at,omitempty"`
// Alerts24h counts the distinct alerts this source refreshed in the last
// day. An alert re-sent every few hours counts once, not once per re-send.
Alerts24h int64 `json:"alerts_24h"`
}
// handleListIntegrations lists a team's integrations with what each has been
// delivering. Never the keys: those exist in plaintext only in the response that
// created them.
func handleListIntegrations(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
@@ -367,28 +477,45 @@ func handleListIntegrations(db *sql.DB) http.HandlerFunc {
return
}
now := time.Now()
rows, err := db.QueryContext(r.Context(), `
SELECT id, team_id, kind, name, created_at, last_used_at
FROM integrations
WHERE team_id = $1
ORDER BY id`, teamID)
SELECT i.id, i.team_id, i.kind, i.name, i.created_at, i.last_used_at,
-- Scalar subqueries, not a join and GROUP BY: each is a
-- single range over alerts_integration_idx, where the join
-- would read every alert a source ever delivered.
(SELECT MAX(received_at) FROM alerts WHERE integration_id = i.id),
(SELECT COUNT(*) FROM alerts
WHERE integration_id = i.id AND received_at >= $2)
FROM integrations i
WHERE i.team_id = $1
ORDER BY i.id`, teamID, now.Add(-sourceQuietAfter).Unix())
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
defer rows.Close()
integrations := []models.Integration{}
integrations := []integrationStatus{}
for rows.Next() {
var i models.Integration
var i integrationStatus
var created int64
var lastUsed *int64
if err := rows.Scan(&i.ID, &i.TeamID, &i.Kind, &i.Name, &created, &lastUsed); err != nil {
var lastUsed, lastAlert *int64
if err := rows.Scan(&i.ID, &i.TeamID, &i.Kind, &i.Name, &created, &lastUsed,
&lastAlert, &i.Alerts24h); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
i.CreatedAt = time.Unix(created, 0).UTC()
i.LastUsedAt = unixPtr(lastUsed)
i.LastAlertAt = unixPtr(lastAlert)
switch {
case i.LastUsedAt == nil:
i.Status = sourceNever
case now.Sub(*i.LastUsedAt) > sourceQuietAfter:
i.Status = sourceQuiet
default:
i.Status = sourceActive
}
integrations = append(integrations, i)
}
if err := rows.Err(); err != nil {
@@ -457,6 +584,54 @@ func handleCreateIntegration(db *sql.DB, publicURL string) http.HandlerFunc {
}
}
// handleRenameIntegration renames a source. The key is untouched, so nothing
// posting with it notices.
func handleRenameIntegration(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
id, err := strconv.ParseInt(chi.URLParam(r, "integrationID"), 10, 64)
if err != nil {
respond(w, http.StatusBadRequest, errResp("invalid integration id"))
return
}
var req struct {
Name string `json:"name"`
}
if err := decodeJSON(r, &req); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid request body"))
return
}
req.Name = strings.TrimSpace(req.Name)
if req.Name == "" {
respond(w, http.StatusBadRequest, errResp("name is required"))
return
}
if len(req.Name) > 100 {
respond(w, http.StatusBadRequest, errResp("name is too long"))
return
}
res, err := db.ExecContext(r.Context(),
"UPDATE integrations SET name = $1 WHERE id = $2 AND team_id = $3", req.Name, id, teamID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if n, _ := res.RowsAffected(); n == 0 {
respond(w, http.StatusNotFound, errResp("not found"))
return
}
w.WriteHeader(http.StatusNoContent)
}
}
func handleDeleteIntegration(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
@@ -493,24 +668,32 @@ func integrationPath(key, kind string) string {
return "/api/integrations/" + key + "/" + kind
}
// teamIDForKey resolves an integration key to its team, and stamps the key's
// alertSource is who an arriving webhook is from: the integration whose key it
// used, and the team that integration puts its alerts in.
type alertSource struct {
integrationID int64
teamID int64
}
// sourceForKey resolves an integration key to its source, and stamps the key's
// last use. An unknown key is not an error worth distinguishing: the caller is
// told nothing beyond "no".
func teamIDForKey(ctx context.Context, db *sql.DB, key string) (int64, error) {
var teamID int64
func sourceForKey(ctx context.Context, db *sql.DB, key string) (alertSource, error) {
var src alertSource
err := db.QueryRowContext(ctx,
"SELECT team_id FROM integrations WHERE key_hash = $1", hashToken(key)).Scan(&teamID)
"SELECT id, team_id FROM integrations WHERE key_hash = $1", hashToken(key)).
Scan(&src.integrationID, &src.teamID)
if errors.Is(err, sql.ErrNoRows) {
return 0, errUnknownIntegration
return alertSource{}, errUnknownIntegration
}
if err != nil {
return 0, err
return alertSource{}, err
}
// Best effort, like an API key's: a failed stamp must not reject an alert.
db.ExecContext(ctx, //nolint:errcheck
"UPDATE integrations SET last_used_at = $1 WHERE key_hash = $2",
time.Now().Unix(), hashToken(key))
return teamID, nil
"UPDATE integrations SET last_used_at = $1 WHERE id = $2",
time.Now().Unix(), src.integrationID)
return src, nil
}
var errUnknownIntegration = errors.New("unknown integration key")
@@ -539,17 +722,24 @@ func defaultTeamID(ctx context.Context, db *sql.DB) (int64, error) {
// A team's dead man's switches
// ---------------------------------------------------------------------------
// deadmanResponse is the wire shape of a team's switch configuration. The
// timeout is seconds rather than a duration string, because that is what the
// column holds and what arithmetic is done on; a client renders it.
type deadmanResponse struct {
TeamID int64 `json:"team_id"`
Matchers string `json:"matchers"`
// deadmanSwitchRequest is what creating a switch takes. The timeout is seconds,
// because that is what the column holds and what arithmetic is done on; a client
// renders it.
type deadmanSwitchRequest struct {
Name string `json:"name"`
Matcher string `json:"matcher"`
TimeoutSeconds int64 `json:"timeout_seconds"`
Severity string `json:"severity"`
}
func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc {
// deadmanSeverities are the severities an incident can open at.
var deadmanSeverities = map[string]bool{"critical": true, "error": true, "warning": true, "info": true}
// handleListTeamDeadman lists a team's switches with what each one's heartbeats
// are doing. A team with none gets an empty list, which is a configuration and
// not an absence: answering 404 would make "off" indistinguishable from "this
// server does not do this".
func handleListTeamDeadman(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
@@ -559,27 +749,26 @@ func handleGetTeamDeadman(db *sql.DB) http.HandlerFunc {
return
}
out := deadmanResponse{TeamID: teamID, Severity: "critical"}
err := db.QueryRowContext(r.Context(),
"SELECT matchers, timeout_seconds, severity FROM deadman_configs WHERE team_id = $1",
teamID).Scan(&out.Matchers, &out.TimeoutSeconds, &out.Severity)
if err != nil && !errors.Is(err, sql.ErrNoRows) {
set, err := deadmanSetForTeam(r.Context(), db, teamID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
out, err := deadmanStatuses(r.Context(), db, teamID, set, time.Now())
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
// A team with no row watches nothing, which is a configuration and not
// an absence: answering 404 would make "off" indistinguishable from
// "this server does not do this".
respond(w, http.StatusOK, out)
}
}
// handleSetTeamDeadman replaces a team's switch configuration.
// handleCreateTeamDeadman adds one switch.
//
// Validated by parsing: a matcher string that survives ParseDeadmanConfig with
// nothing usable in it is rejected rather than stored, because a switch that
// silently watches nothing is the failure this feature exists to prevent.
func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc {
// Validated by parsing: a matcher with no alertname is rejected rather than
// stored, because a switch that silently watches nothing is the failure this
// feature exists to prevent.
func handleCreateTeamDeadman(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
@@ -589,50 +778,87 @@ func handleSetTeamDeadman(db *sql.DB) http.HandlerFunc {
return
}
var req struct {
Matchers string `json:"matchers"`
TimeoutSeconds int64 `json:"timeout_seconds"`
Severity string `json:"severity"`
}
var req deadmanSwitchRequest
if err := decodeJSON(r, &req); err != nil {
respond(w, http.StatusBadRequest, errResp("invalid request body"))
return
}
req.Matchers = strings.TrimSpace(req.Matchers)
req.Matcher = strings.TrimSpace(req.Matcher)
req.Name = strings.TrimSpace(req.Name)
if req.Severity == "" {
req.Severity = "critical"
}
if req.TimeoutSeconds < 0 {
respond(w, http.StatusBadRequest, errResp("timeout_seconds must not be negative"))
if !deadmanSeverities[req.Severity] {
respond(w, http.StatusBadRequest, errResp("severity must be critical, error, warning or info"))
return
}
if req.Matchers != "" {
parsed := parseDeadmanQuietly(req.Matchers, time.Duration(req.TimeoutSeconds)*time.Second, req.Severity)
if len(parsed.Matchers) == 0 {
if req.TimeoutSeconds <= 0 {
respond(w, http.StatusBadRequest, errResp("timeout_seconds must be positive"))
return
}
if strings.Contains(req.Matcher, ";") {
respond(w, http.StatusBadRequest, errResp("one matcher per switch: add another switch instead of separating with ;"))
return
}
m, err := parseDeadmanMatcher(req.Matcher)
if err != nil {
respond(w, http.StatusBadRequest, errResp(
"no usable matchers: each must name an alertname, as in alertname=Watchdog,cluster=prod"))
"unusable matcher ("+err.Error()+"): each must name an alertname, as in alertname=Watchdog,cluster=prod"))
return
}
if req.Name == "" {
req.Name = m.config()
}
if len(req.Name) > 100 {
respond(w, http.StatusBadRequest, errResp("name is too long"))
return
}
if _, err := db.ExecContext(r.Context(), `
INSERT INTO deadman_configs (team_id, matchers, timeout_seconds, severity, updated_at)
VALUES ($1, $2, $3, $4, `+nowEpoch+`)
ON CONFLICT (team_id) DO UPDATE SET
matchers = excluded.matchers,
timeout_seconds = excluded.timeout_seconds,
severity = excluded.severity,
updated_at = excluded.updated_at`,
teamID, req.Matchers, req.TimeoutSeconds, req.Severity); err != nil {
var id int64
if err := db.QueryRowContext(r.Context(), `
INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
VALUES ($1, $2, $3, $4, $5) RETURNING id`,
teamID, req.Name, m.config(), req.TimeoutSeconds, req.Severity).Scan(&id); err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
respond(w, http.StatusOK, deadmanResponse{
TeamID: teamID,
Matchers: req.Matchers,
TimeoutSeconds: req.TimeoutSeconds,
Severity: req.Severity,
respond(w, http.StatusCreated, deadmanSwitchStatus{
ID: id, Name: req.Name, Matcher: m.config(),
TimeoutSeconds: req.TimeoutSeconds, Severity: req.Severity,
Status: switchDormant, Sources: []deadmanSource{},
})
}
}
// handleDeleteTeamDeadman removes a switch. An incident it already opened stays
// open until somebody resolves it: deleting the switch says "stop watching", not
// "the problem is gone".
func handleDeleteTeamDeadman(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
teamID, ok := teamParam(w, r)
if !ok {
return
}
if !requireTeamOwner(w, r, teamID) {
return
}
switchID, err := strconv.ParseInt(chi.URLParam(r, "switchID"), 10, 64)
if err != nil {
respond(w, http.StatusBadRequest, errResp("invalid switch id"))
return
}
res, err := db.ExecContext(r.Context(),
"DELETE FROM deadman_switches WHERE id = $1 AND team_id = $2", switchID, teamID)
if err != nil {
respond(w, http.StatusInternalServerError, errResp("internal error"))
return
}
if n, _ := res.RowsAffected(); n == 0 {
respond(w, http.StatusNotFound, errResp("switch not found"))
return
}
w.WriteHeader(http.StatusNoContent)
}
}
+24 -2
View File
@@ -5,6 +5,7 @@ import (
"embed"
"fmt"
"io/fs"
"log"
"sort"
"strings"
"time"
@@ -15,6 +16,19 @@ import (
//go:embed migrations
var migrationsFS embed.FS
// pingAttempts and pingRetryDelay bound the retry on the first connection.
// This pod's own IP can reach the Postgres pod's node before that node's
// NetworkPolicy enforcement (kube-router, reacting to the pod's creation
// event) has added it to the allowed-source set, which fails the ping with
// "connection refused" rather than a timeout. That race resolves within
// several seconds in practice; five attempts two seconds apart give it
// comfortable room without turning a genuinely absent database into a long
// hang.
const (
pingAttempts = 5
pingRetryDelay = 2 * time.Second
)
// Open connects to Postgres. dsn is a libpq connection string or URL, e.g.
// postgres://terdut:secret@localhost:5432/terdut?sslmode=disable.
//
@@ -33,11 +47,19 @@ func Open(dsn string) (*sql.DB, error) {
db.SetMaxOpenConns(10)
db.SetMaxIdleConns(5)
db.SetConnMaxLifetime(time.Hour)
if err := db.Ping(); err != nil {
for attempt := 1; ; attempt++ {
err = db.Ping()
if err == nil {
return db, nil
}
if attempt == pingAttempts {
db.Close()
return nil, fmt.Errorf("ping: %w", err)
}
return db, nil
log.Printf("open db: ping attempt %d/%d failed, retrying in %s: %v", attempt, pingAttempts, pingRetryDelay, err)
time.Sleep(pingRetryDelay)
}
}
// Migrate applies every embedded migration that has not been applied yet, in
@@ -0,0 +1,23 @@
-- Similar incidents: a signature per incident, so "has this happened before"
-- is an indexed equality instead of a search.
--
-- The signature is the alert name plus the group labels that identify WHAT is
-- broken, minus the ones that only say WHERE it happened to run this time
-- (instance, pod, ...). Two incidents with the same signature in the same team
-- are the same problem for a responder's purposes.
--
-- Computed in Go for new incidents (incidentSignature in incident_store.go).
-- The backfill below MUST produce the same string; keep the volatile list in
-- both places in step.
ALTER TABLE incidents ADD COLUMN signature TEXT NOT NULL DEFAULT '';
UPDATE incidents SET signature =
COALESCE(NULLIF(group_labels->>'alertname', ''), title) || '|' ||
COALESCE((
SELECT string_agg(e.k || '=' || e.v, ',' ORDER BY e.k)
FROM jsonb_each_text(incidents.group_labels) AS e(k, v)
WHERE e.k <> 'alertname'
AND e.k NOT IN ('instance', 'pod', 'pod_name', 'pod_ip', 'container', 'container_name', 'endpoint')
), '');
CREATE INDEX incidents_signature_idx ON incidents(team_id, signature, triggered_at DESC);
@@ -0,0 +1,54 @@
-- Dead man's switches become rows of their own.
--
-- 004 kept a team's switches in one string with one timeout and one severity,
-- which was enough to configure them and not enough to show them: there was no
-- thing to list, nothing to hang a status on, and every switch in a team had to
-- share a deadline. A row per switch gives each its own name, matcher, timeout
-- and severity, and gives the Team → Switches page something to be a list of.
--
-- The matcher keeps the syntax the string used, one matcher per row:
-- `alertname=Watchdog,cluster=prod`. The unit of monitoring is still the
-- fingerprint, so a matcher that many clusters satisfy is still one switch row
-- watching several independent heartbeats.
CREATE TABLE deadman_switches (
id BIGSERIAL PRIMARY KEY,
team_id BIGINT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
-- What the owner calls it. Defaults to the matcher when they do not say.
name TEXT NOT NULL,
-- "," separates the label conditions, "=" is exact equality, and alertname is
-- mandatory: it is what keeps the sweeper's candidate query on an index.
matcher TEXT NOT NULL,
-- Seconds of silence before the switch is declared dead. Never zero: a switch
-- that cannot fire is deleted, not disabled.
timeout_seconds BIGINT NOT NULL CHECK (timeout_seconds > 0),
-- The severity its incidents open at. See 004 for why they carry their own.
severity TEXT NOT NULL DEFAULT 'critical',
created_at BIGINT NOT NULL DEFAULT FLOOR(EXTRACT(EPOCH FROM now()))::bigint
);
CREATE INDEX deadman_switches_team_idx ON deadman_switches (team_id);
-- Carry every team's configuration over, one row per matcher. A team whose
-- timeout was zero had switches turned off, which is now "no rows".
INSERT INTO deadman_switches (team_id, name, matcher, timeout_seconds, severity)
SELECT c.team_id, btrim(m), btrim(m), c.timeout_seconds, c.severity
FROM deadman_configs c,
LATERAL regexp_split_to_table(c.matchers, ';') AS m
WHERE c.timeout_seconds > 0
AND btrim(m) <> ''
ORDER BY c.team_id;
-- The server seeds environment defaults into teams once, and remembers that it
-- did. An install that had a row per team was already seeded; without this
-- marker the first start after upgrading would seed teams that had switched
-- theirs off.
INSERT INTO settings (key, value)
SELECT 'deadman_seeded', '1'
WHERE EXISTS (SELECT 1 FROM deadman_configs);
DROP TABLE deadman_configs;
@@ -0,0 +1,21 @@
-- Which alert source an alert last arrived on.
--
-- Team -> Sources shows when each source last posted, which integrations
-- already knew (last_used_at, stamped on every webhook). What it could not say
-- was what a source delivered: an alert never recorded the key it came in on, so
-- "prod alertmanager" and "staging alertmanager" were indistinguishable once
-- inside. This column is that link, and lets the page show each source's last
-- alert and how many alerts it has kept fresh over the past day.
--
-- Last sender wins: every accepted payload restamps it, the way it advances
-- received_at. Two sources posting the same fingerprint into one team is
-- already one alert, and it is attributed to whichever spoke last.
--
-- Nullable, and not backfilled. Alerts that arrived before this migration have
-- no source, and NULL says so honestly rather than guessing. It heals by itself:
-- Alertmanager re-sends every alert each repeat_interval, and each re-send is an
-- accepted payload. Deleting a source keeps its alerts, unattributed.
ALTER TABLE alerts ADD COLUMN integration_id BIGINT REFERENCES integrations(id) ON DELETE SET NULL;
CREATE INDEX alerts_integration_idx ON alerts (integration_id, received_at)
WHERE integration_id IS NOT NULL;
+12
View File
@@ -79,3 +79,15 @@ type IncidentEvent struct {
Detail *string `json:"detail,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
// SimilarIncident is an earlier, resolved incident with the same signature as
// the one being looked at. ResolutionNotes are the "what fixed it" notes;
// NoteCount counts the plain working notes, which live on the timeline.
type SimilarIncident struct {
ID int64 `json:"id"`
Title string `json:"title"`
TriggeredAt time.Time `json:"triggered_at"`
ResolvedAt time.Time `json:"resolved_at"`
NoteCount int `json:"note_count"`
ResolutionNotes []IncidentEvent `json:"resolution_notes"`
}
+26
View File
@@ -0,0 +1,26 @@
package web
import (
"io/fs"
"strings"
"testing"
)
func TestCopyIncidentIsEmbedded(t *testing.T) {
sub, err := fs.Sub(files, "static")
if err != nil {
t.Fatal(err)
}
for file, want := range map[string]string{
"js/incident.js": "copyIncident",
"js/ui.js": "copy:",
} {
b, err := fs.ReadFile(sub, file)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(b), want) {
t.Errorf("%s lacks %s", file, want)
}
}
}
+109 -25
View File
@@ -48,7 +48,11 @@
--mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
--topbar-h: 52px;
--tabbar-h: 58px;
/* No bottom tab bar on any breakpoint any more — mobile uses the hamburger
menu in the topbar, desktop the sidebar — so this stays 0. Kept as a
variable rather than deleted since .view, .toast and .nav's own height
calc still read it. */
--tabbar-h: 0px;
--safe-top: env(safe-area-inset-top, 0px);
--safe-bottom: env(safe-area-inset-bottom, 0px);
}
@@ -169,7 +173,7 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
transition: background 0.12s, border-color 0.12s, opacity 0.12s;
}
.btn:hover { background: var(--surface-hover); }
.btn:disabled { opacity: 0.55; cursor: default; }
.btn:disabled, .btn-sm:disabled { opacity: 0.55; cursor: default; }
.btn-primary { background: var(--accent); border-color: var(--accent); color: var(--accent-text); }
.btn-primary:hover { background: var(--accent); filter: brightness(1.06); }
.btn-danger { background: var(--crit); border-color: var(--crit); color: #fff; }
@@ -196,7 +200,13 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
-webkit-backdrop-filter: saturate(1.4) blur(12px);
border-bottom: 1px solid var(--border);
}
.topbar-title { font-size: 18px; font-weight: 700; letter-spacing: -0.01em; }
.topbar-left { display: flex; align-items: center; gap: 8px; min-width: 0; }
.topbar-title {
font-size: 18px; font-weight: 700; letter-spacing: -0.01em;
overflow: hidden; text-overflow: ellipsis; white-space: nowrap; min-width: 0;
}
#menu-btn { position: relative; }
.nav-badge.menu-btn-badge { top: 2px; left: auto; right: 2px; }
.open-pill {
display: inline-flex; align-items: center; gap: 6px;
@@ -209,15 +219,12 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
.open-pill.has-triggered::before { background: var(--crit); }
.open-pill.all-acked::before { background: var(--warn); }
/* Bottom tab bar on phones. */
/* Hidden on phones — mobile navigates through the hamburger menu in the
topbar instead (see #menu-btn / openNavMenu in app.js). Reappears as the
left sidebar from 900px, where the desktop block below redeclares display. */
.nav {
display: none;
position: fixed; left: 0; right: 0; bottom: 0; z-index: 20;
/* One column per link, however many there are. This was repeat(4, 1fr) when
there were four tabs; Team and Admin arriving pushed six items into four
columns, which on a phone is how they stopped fitting. Auto columns mean
the next tab cannot break the row either — and Admin is only rendered for
an administrator, so the count genuinely varies between viewers. */
display: grid; grid-auto-flow: column; grid-auto-columns: 1fr;
height: calc(var(--tabbar-h) + var(--safe-bottom));
padding-bottom: var(--safe-bottom);
background: color-mix(in srgb, var(--surface) 92%, transparent);
@@ -239,16 +246,6 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
}
.nav-link svg { width: 24px; height: 24px; flex: none; fill: none; stroke: currentColor; stroke-width: 1.8; stroke-linecap: round; stroke-linejoin: round; }
/* Narrow phones, where six tabs each get about 55-65px. Tightening is enough:
the widest label, "On-call", is about 38px at this size, so nothing has to
be hidden. The ellipsis above is the backstop if a future tab is named
something longer. */
@media (max-width: 420px) {
.nav-link { font-size: 10px; gap: 1px; }
.nav-link svg { width: 21px; height: 21px; }
.nav-badge { left: calc(50% + 4px); min-width: 16px; height: 16px; font-size: 10px; line-height: 16px; }
}
.nav-link[aria-current="page"] { color: var(--accent); }
.nav-badge {
position: absolute; top: 6px; left: calc(50% + 6px);
@@ -362,7 +359,16 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
.badge.st-triggered, .badge.st-firing { background: var(--crit-soft); color: var(--crit); }
.badge.st-acknowledged { background: var(--warn-soft); color: var(--warn); }
.badge.st-snoozed { background: var(--snooze-soft); color: var(--snooze); }
.badge.st-resolved { background: var(--ok-soft); color: var(--ok); }
.badge.st-resolved, .badge.st-healthy { background: var(--ok-soft); color: var(--ok); }
.badge.st-dead { background: var(--crit-soft); color: var(--crit); }
/* Dormant is the plain badge on purpose: nothing has gone wrong and nothing has
gone right, which is what the muted default already says. */
.badge.st-dormant, .badge.st-never { background: var(--surface-2); color: var(--muted); }
.badge.st-active { background: var(--ok-soft); color: var(--ok); }
.badge.st-quiet, .badge.st-escalating { background: var(--warn-soft); color: var(--warn); }
.badge.st-ready, .badge.st-oncall { background: var(--ok-soft); color: var(--ok); }
.badge.st-reachable { background: var(--surface-2); color: var(--muted); }
.badge.st-unreachable, .badge.st-unpageable { background: var(--crit-soft); color: var(--crit); }
.badge.sev-critical { background: var(--crit-soft); color: var(--crit); }
.badge.sev-warning { background: var(--warn-soft); color: var(--warn); }
.badge.sev-info { background: var(--info-soft); color: var(--info); }
@@ -380,6 +386,8 @@ input:focus, textarea:focus { outline: none; border-color: var(--accent); box-sh
-webkit-backdrop-filter: saturate(1.4) blur(12px);
border-bottom: 1px solid var(--border);
}
.detail-head .copy { margin-left: auto; }
.clip-buffer { position: fixed; top: 0; left: 0; opacity: 0; pointer-events: none; }
.detail-head .crumb { font-weight: 600; color: var(--muted); font-size: 14px; }
.detail-title { font-size: 21px; font-weight: 750; letter-spacing: -0.01em; margin: 16px 0 8px; overflow-wrap: anywhere; }
.detail-badges { display: flex; flex-wrap: wrap; gap: 6px; margin-bottom: 14px; }
@@ -453,11 +461,16 @@ details[open] > summary { margin-bottom: 8px; }
.tl-text { overflow-wrap: anywhere; }
.tl-text .who { font-weight: 650; }
.tl-time { color: var(--faint); font-size: 12px; }
.tl-note .note {
.note {
margin-top: 6px; padding: 10px 12px;
background: var(--surface-2); border-radius: var(--radius-sm);
white-space: pre-wrap; overflow-wrap: anywhere;
}
.note-fix { background: var(--ok-soft); border-left: 3px solid var(--ok); }
.similar { list-style: none; margin: 0; padding: 0; }
.similar-item { padding: 10px 0; }
.similar-item + .similar-item { border-top: 1px solid var(--border, var(--surface-2)); }
.check { display: flex; align-items: center; gap: 8px; font-size: 14px; color: var(--muted); }
.note-actions { display: flex; justify-content: flex-end; }
.note-actions .btn { color: var(--muted); }
@@ -601,8 +614,6 @@ kbd {
/* ---------- desktop ---------- */
@media (min-width: 900px) {
:root { --tabbar-h: 0px; }
.app { display: grid; grid-template-columns: 220px 1fr; height: 100dvh; }
.nav {
@@ -638,6 +649,10 @@ kbd {
.view-queue .pane { overflow: auto; height: 100dvh; }
.pane-list { border-right: 1px solid var(--border); }
.pane-list .chips { position: sticky; top: 0; z-index: 2; background: var(--bg); padding-top: 16px; }
/* The pane is 340-420px wide and a mouse cannot scroll a row whose scrollbar
is hidden, so the chips wrap here instead: Archived stays reachable. */
.pane-list .chips { flex-wrap: wrap; overflow-x: visible; }
.pane-list .chip-sep { display: none; }
.view-queue:not(.has-detail) .pane-detail { display: block; }
/* On desktop the list stays visible next to the detail. */
@@ -685,6 +700,18 @@ kbd {
.disabled-row td { opacity: 0.55; }
.btn-sm.danger { color: var(--crit); border-color: var(--crit-soft); }
/* --- status lists: alert sources and dead man's switches -----------------
Six columns do not fit a phone, so the table scrolls inside its card rather
than the page. A heartbeat under a switch with several is indented, the way
the escalation ladder indents its levels. */
.card-head { display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap; }
.table-scroll { overflow-x: auto; margin-top: 12px; }
.status-table th, .status-table td { white-space: nowrap; }
.status-table td.wrap { white-space: normal; min-width: 12em; }
.status-table .source-row td { border-bottom-style: dashed; }
.status-table .source-row td:first-child { padding-left: 16px; }
.source-labels { display: flex; flex-wrap: wrap; gap: 4px; align-items: center; }
.inline-form { display: flex; gap: 8px; margin-top: 12px; }
.inline-form input { flex: 1; min-width: 0; }
@@ -746,7 +773,7 @@ kbd {
too alike down a column to read, so a day carries an initial in that
person's colour and the legend underneath says whose. A shift is then a run
of one colour, which is the shape the question actually has. */
.rota-grid { display: grid; grid-template-columns: repeat(7, 1fr); gap: 2px; padding: 10px; }
.rota-grid { display: grid; grid-template-columns: 2.4em repeat(7, 1fr); gap: 2px; padding: 10px; }
.rota-wd {
padding-bottom: 4px; text-align: center;
color: var(--muted); font-size: 11px; font-weight: 700;
@@ -760,6 +787,21 @@ kbd {
}
button.rota-day { cursor: pointer; }
button.rota-day:hover { background: var(--surface-2); }
/* The week number starts each row. Quiet by default, because it is a label
first; an owner's tap on it is the second thing it does. */
.rota-week {
display: grid; place-items: center;
border: 0; border-radius: var(--radius-sm); background: none;
font: inherit; font-size: 12px; font-variant-numeric: tabular-nums;
color: var(--faint);
}
button.rota-week { cursor: pointer; }
button.rota-week:hover { background: var(--surface-2); color: var(--text); }
.rota-week.current { color: var(--accent); font-weight: 700; }
/* The sheet's row of who holds each day of the week. */
.week-holders { display: flex; justify-content: space-between; gap: 4px; margin: 4px 0 12px; }
.week-holder { display: flex; flex-direction: column; align-items: center; gap: 4px; flex: 1; }
.week-holder.past { opacity: 0.55; }
.rota-num { color: var(--muted); font-size: 12px; font-variant-numeric: tabular-nums; }
.rota-day.today { background: var(--accent-soft); }
.rota-day.today .rota-num { color: var(--accent); font-weight: 700; }
@@ -803,6 +845,11 @@ button.rota-day:hover { background: var(--surface-2); }
.ladder-head { display: flex; align-items: center; gap: 10px; margin-bottom: 6px; }
.ladder-targets { display: flex; flex-direction: column; gap: 6px; margin-top: 8px; }
.target-row { display: flex; gap: 6px; align-items: center; flex-wrap: wrap; }
.ladder-editor { display: flex; flex-direction: column; gap: 10px; align-items: flex-start; margin-top: 12px; }
/* A target that would not wake anybody says why, in place: it is the reason a
level is red, and the thing to go and fix. */
.target-line { display: flex; gap: 8px; align-items: baseline; flex-wrap: wrap; }
.target-problem { color: var(--crit); font-size: 12px; font-weight: 600; }
/* An integration key is shown exactly once, so it should look like something
to act on rather than another row of text. */
@@ -872,3 +919,40 @@ button.rota-day:hover { background: var(--surface-2); }
.overview-head { display: flex; align-items: baseline; gap: 8px; }
.overview-count { margin-left: auto; color: var(--muted); font-size: 18px; font-weight: 700; }
.overview-item p { margin: 4px 0 0; }
/* ---------- stats page ---------- */
#view-stats .chips { padding-left: 0; padding-right: 0; }
.stats { display: grid; gap: 14px; padding-bottom: 16px; }
.stat-tiles { display: grid; grid-template-columns: repeat(2, 1fr); gap: 8px; }
.stat-tile {
--sev: var(--border-strong);
background: var(--surface); border: 1px solid var(--border);
border-left: 3px solid var(--sev); border-radius: var(--radius);
box-shadow: var(--shadow); padding: 12px;
}
.stat-tile.st-triggered { --sev: var(--crit); }
.stat-tile.st-acknowledged { --sev: var(--warn); }
.stat-tile.st-resolved { --sev: var(--ok); }
.stat-value { font-size: 24px; font-weight: 700; font-variant-numeric: tabular-nums; }
.stat-label { margin-top: 2px; font-size: 12px; color: var(--muted); }
.chart-card + .chart-card { margin-top: 0; }
.chart-title {
margin-bottom: 10px; font-size: 13px; font-weight: 700;
text-transform: uppercase; letter-spacing: 0.06em; color: var(--muted);
}
.hbars { list-style: none; display: grid; gap: 6px; }
.hbar { display: grid; grid-template-columns: minmax(80px, 34%) 1fr auto; align-items: center; gap: 8px; font-size: 13px; }
.hbar-name { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.hbar-track { height: 10px; border-radius: 5px; background: var(--surface-2); overflow: hidden; }
.hbar-fill { display: block; height: 100%; border-radius: 5px; background: var(--ok); }
.hbar-count { min-width: 2ch; text-align: right; color: var(--muted); font-variant-numeric: tabular-nums; }
.columns { display: block; width: 100%; height: auto; }
.columns .axis { stroke: var(--border-strong); stroke-width: 1; }
.columns .col-hit { fill: transparent; }
.columns .col-bar { fill: var(--accent); }
.columns .col:hover .col-bar { opacity: 0.75; }
.columns .col-label { fill: var(--faint); font-size: 10px; font-family: var(--font); }
@media (min-width: 900px) {
.stat-tiles { grid-template-columns: repeat(3, 1fr); }
}
+11
View File
@@ -90,6 +90,10 @@
<svg viewBox="0 0 24 24" aria-hidden="true"><path d="M6 16V11a6 6 0 0 1 12 0v5l1.5 2h-15z"/><path d="M10 20.5a2 2 0 0 0 4 0"/></svg>
<span class="nav-label">Alerts</span>
</a>
<a class="nav-link" href="/stats" data-section="stats" aria-label="Stats">
<svg viewBox="0 0 24 24" aria-hidden="true"><path d="M4 20h16M7 20v-7M12 20V6M17 20v-10"/></svg>
<span class="nav-label">Stats</span>
</a>
<a class="nav-link" href="/team" data-section="team" aria-label="Team">
<svg viewBox="0 0 24 24" aria-hidden="true"><circle cx="9" cy="8" r="3"/><circle cx="17" cy="9" r="2.5"/><path d="M3 19a6 6 0 0 1 12 0M15 19a5 5 0 0 1 6-4"/></svg>
<span class="nav-label">Team</span>
@@ -108,7 +112,13 @@
</nav>
<header class="topbar">
<div class="topbar-left">
<button class="btn btn-ghost btn-icon" id="menu-btn" type="button" aria-label="Menu" aria-haspopup="menu">
<svg viewBox="0 0 24 24" aria-hidden="true"><path d="M4 7h16M4 12h16M4 17h16"/></svg>
<span class="nav-badge menu-btn-badge" data-badge hidden></span>
</button>
<h1 class="topbar-title" id="topbar-title">Queue</h1>
</div>
<span class="open-pill" id="open-pill" hidden></span>
</header>
@@ -122,6 +132,7 @@
<section id="view-oncall" class="view view-page" data-view="oncall" hidden></section>
<section id="view-alerts" class="view view-page" data-view="alerts" hidden></section>
<section id="view-stats" class="view view-page" data-view="stats" hidden></section>
<section id="view-team" class="view view-page" data-view="team" hidden></section>
<section id="view-admin" class="view view-page" data-view="admin" hidden></section>
<!-- One person, at /admin/users/{id}: reached from the Admin tab's user
-1
View File
@@ -35,7 +35,6 @@ function render() {
h('div', { class: 'page-head' }),
h('button', { class: 'btn btn-block', type: 'button', onclick: signOut }, icon('logout'), 'Sign out'),
h('p', { class: 'foot-note', text: 'Statistics are in terdut-tui for now.' }),
);
}
+16 -4
View File
@@ -83,15 +83,22 @@ export const timeline = (id) => call('GET', `/incidents/${id}/timeline`);
export const acknowledge = (id) => call('POST', `/incidents/${id}/acknowledge`);
export const unacknowledge = (id) => call('DELETE', `/incidents/${id}/acknowledge`);
export const resolve = (id) => call('POST', `/incidents/${id}/resolve`);
export const resolve = (id, resolution) => call('POST', `/incidents/${id}/resolve`, resolution ? { body: { resolution } } : {});
export const assign = (id, userID) => call('POST', `/incidents/${id}/assign`, { body: { user_id: userID } });
export const snooze = (id, spec) => call('POST', `/incidents/${id}/snooze`, { body: spec });
export const unsnooze = (id) => call('DELETE', `/incidents/${id}/snooze`);
export const archive = (id) => call('POST', `/incidents/${id}/archive`);
export const unarchive = (id) => call('DELETE', `/incidents/${id}/archive`);
export const addNote = (id, content) => call('POST', `/incidents/${id}/notes`, { body: { content } });
export const addNote = (id, content, pinned = false) => call('POST', `/incidents/${id}/notes`, { body: { content, pinned } });
export const similar = (id) => call('GET', `/incidents/${id}/similar`);
export const deleteNote = (id, eventID) => call('DELETE', `/incidents/${id}/notes/${eventID}`);
// stats
export const statsIncidents = (query) => call('GET', '/stats/incidents', { query });
export const statsTop = (query) => call('GET', '/stats/alerts/top', { query });
export const statsByHour = (query) => call('GET', '/stats/alerts/by-hour', { query });
export const statsByDay = (query) => call('GET', '/stats/alerts/by-day', { query });
// alerts
export const alerts = (query, opts) => call('GET', '/alerts', { query, ...opts });
@@ -126,11 +133,16 @@ export const removeTeamMember = (id, userID) => call('DELETE', `/teams/${id}/mem
export const integrations = (id) => call('GET', `/teams/${id}/integrations`);
export const createIntegration = (id, name) =>
call('POST', `/teams/${id}/integrations`, { body: { name } });
export const renameIntegration = (id, integrationID, name) =>
call('PATCH', `/teams/${id}/integrations/${integrationID}`, { body: { name } });
export const deleteIntegration = (id, integrationID) =>
call('DELETE', `/teams/${id}/integrations/${integrationID}`);
export const deadman = (id) => call('GET', `/teams/${id}/deadman`);
export const setDeadman = (id, body) => call('PUT', `/teams/${id}/deadman`, { body });
export const deadmanSwitches = (id) => call('GET', `/teams/${id}/deadman/switches`);
export const createDeadmanSwitch = (id, body) =>
call('POST', `/teams/${id}/deadman/switches`, { body });
export const deleteDeadmanSwitch = (id, switchID) =>
call('DELETE', `/teams/${id}/deadman/switches/${switchID}`);
export const escalation = (id) => call('GET', `/teams/${id}/escalation`);
export const setEscalation = (id, body) => call('PUT', `/teams/${id}/escalation`, { body });
+49 -2
View File
@@ -8,6 +8,7 @@ import * as queue from './queue.js';
import * as incident from './incident.js';
import * as oncall from './oncall.js';
import * as alerts from './alerts.js';
import * as stats from './stats.js';
import * as account from './account.js';
import * as team from './team.js';
import * as admin from './admin.js';
@@ -23,6 +24,7 @@ const SECTIONS = {
queue: { title: 'Queue', view: queue },
oncall: { title: 'On-call', view: oncall },
alerts: { title: 'Alerts', view: alerts },
stats: { title: 'Stats', view: stats },
team: { title: 'Team', view: team },
admin: { title: 'Admin', view: admin },
adminuser: { title: 'User', view: adminuser, nav: 'admin' },
@@ -30,6 +32,18 @@ const SECTIONS = {
more: { title: 'Account', view: account },
};
// The mobile hamburger menu's contents — the same sections the desktop
// sidebar's .nav-link list carries in index.html, in the same order.
const NAV_ITEMS = [
{ path: '/', section: 'queue', label: 'Queue', icon: 'queueList' },
{ path: '/oncall', section: 'oncall', label: 'On-call', icon: 'calendar' },
{ path: '/alerts', section: 'alerts', label: 'Alerts', icon: 'bell' },
{ path: '/stats', section: 'stats', label: 'Stats', icon: 'chart' },
{ path: '/team', section: 'team', label: 'Team', icon: 'team' },
{ path: '/admin', section: 'admin', label: 'Admin', icon: 'shield', adminOnly: true },
{ path: '/more', section: 'more', label: 'Account', icon: 'user' },
];
function parseRoute(pathname) {
const m = pathname.match(/^\/incidents\/(\d+)\/?$/);
if (m) return { section: 'queue', incident: Number(m[1]) };
@@ -47,7 +61,7 @@ function parseRoute(pathname) {
if (t) return { section: 'admin', tab: t.tab };
const tt = team.TABS.find((x) => x.path === `/${name}`);
if (tt) return { section: 'team', tab: tt.tab };
if (name === 'oncall' || name === 'alerts' || name === 'more') return { section: name };
if (name === 'oncall' || name === 'alerts' || name === 'stats' || name === 'more') return { section: name };
return { section: 'queue', incident: null };
}
@@ -132,6 +146,35 @@ function render() {
updateTitle();
}
// ---------- nav menu ----------
// The mobile hamburger menu: same shape as the sheet-based action menus in
// incident.js (openSheet + a <ul class="menu"> of menu-item buttons), one
// item per NAV_ITEMS entry, resolving with a path for navigate() to use.
function openNavMenu() {
const current = SECTIONS[route.section].nav || route.section;
const triggered = state.open.filter((i) => i.status === 'triggered').length;
const items = NAV_ITEMS.filter((n) => !n.adminOnly || state.me?.user?.is_admin);
ui.openSheet(() => [
ui.h('h2', { class: 'sheet-title', text: 'Sections' }),
ui.h('ul', { class: 'menu', role: 'menu' }, items.map((n) =>
ui.h('li', {}, ui.h('button', {
class: 'menu-item',
type: 'button',
role: 'menuitemradio',
'aria-checked': String(n.section === current),
onclick: () => ui.closeSheet(n.path),
},
ui.icon(n.icon),
n.label,
n.section === 'queue' && triggered > 0 && ui.badge(String(triggered), 'st-triggered menu-sub'),
))),
),
]).then((path) => {
if (path) navigate(path);
});
}
// ---------- refresh + badges ----------
async function refresh() {
@@ -158,9 +201,12 @@ function updateBadges() {
pill.classList.toggle('has-triggered', triggered > 0);
pill.classList.toggle('all-acked', open > 0 && triggered === 0);
const badge = document.querySelector('[data-badge]');
// Two badges carry this count: the sidebar's Queue tab (desktop) and the
// hamburger button (mobile) — only one of the two is ever visible at once.
for (const badge of document.querySelectorAll('[data-badge]')) {
badge.hidden = triggered === 0;
badge.textContent = String(triggered);
}
updateTitle();
}
@@ -180,6 +226,7 @@ async function boot() {
document.addEventListener('keydown', onKey);
$('login-form').addEventListener('submit', onLogin);
$('signup-form').addEventListener('submit', onSignup);
$('menu-btn').addEventListener('click', openNavMenu);
// /signup is the one route that works without a session.
if (location.pathname.replace(/\/$/, '') === '/signup') {
+10 -9
View File
@@ -66,21 +66,22 @@ export function mondayOf(d) {
return r;
}
// ISO 8601 week number: weeks start on Monday and week 1 is the one holding the
// year's first Thursday, which is what a rota that runs Monday to Sunday means
// by "week 40". Taken from the Thursday of d's week, whose year is the week's.
export function isoWeek(d) {
const thu = new Date(d.getFullYear(), d.getMonth(), d.getDate());
thu.setDate(thu.getDate() + 3 - ((thu.getDay() + 6) % 7));
const jan4 = new Date(thu.getFullYear(), 0, 4);
return 1 + Math.round(((thu - jan4) / 86400000 - 3 + ((jan4.getDay() + 6) % 7)) / 7);
}
export function addDays(d, n) {
const r = new Date(d);
r.setDate(r.getDate() + n);
return r;
}
// ISO 8601 week number.
export function isoWeek(d) {
const t = new Date(Date.UTC(d.getFullYear(), d.getMonth(), d.getDate()));
const day = t.getUTCDay() || 7;
t.setUTCDate(t.getUTCDate() + 4 - day);
const yearStart = new Date(Date.UTC(t.getUTCFullYear(), 0, 1));
return Math.ceil(((t - yearStart) / DAY + 1) / 7);
}
export const STATUS_LABEL = {
triggered: 'Triggered',
acknowledged: 'Acknowledged',
+160 -12
View File
@@ -17,6 +17,7 @@ const pane = () => document.getElementById('detail');
let currentID = null;
let inc = null;
let events = [];
let similarList = [];
let error = null;
let busy = false;
@@ -38,10 +39,15 @@ export async function refresh() {
const id = currentID;
if (id == null) return;
try {
const [i, t] = await Promise.all([api.incident(id), api.timeline(id)]);
// Similar incidents are a courtesy: an older server answers 404 and a
// failure here must not hide the incident itself.
const [i, t, sim] = await Promise.all([
api.incident(id), api.timeline(id), api.similar(id).catch(() => []),
]);
if (id !== currentID) return;
inc = i;
events = t;
similarList = sim;
error = null;
} catch (err) {
if (id !== currentID) return;
@@ -60,6 +66,8 @@ function render() {
h('button', { class: 'btn btn-ghost btn-icon back', type: 'button', 'aria-label': 'Back to queue', onclick: back },
icon('back')),
h('span', { class: 'crumb', text: currentID != null ? `Incident #${currentID}` : '' }),
inc && h('button', { class: 'btn btn-ghost btn-icon copy', type: 'button', 'aria-label': 'Copy incident', title: 'Copy incident (y)', onclick: copyIncident },
icon('copy')),
);
if (!inc) {
@@ -81,6 +89,7 @@ function render() {
facts(),
groupLabels(),
alertsSection(),
similarSection(),
timelineSection(),
),
actionBar(),
@@ -179,8 +188,9 @@ function alertItem(a) {
// ---------- timeline ----------
function eventText(ev) {
const person = ev.user_id != null ? who(ev.user_id, ev.username) : null;
// named spells users out instead of "you", for text that leaves this page.
function eventText(ev, named = false) {
const person = ev.user_id != null ? (named ? ev.username || 'someone' : who(ev.user_id, ev.username)) : null;
const strong = (t) => h('span', { class: 'who', text: t || 'someone' });
const alertName = () => {
const a = (inc.alerts || []).find((x) => x.id === ev.alert_id);
@@ -197,6 +207,7 @@ function eventText(ev) {
case 'unsnoozed': return [strong(person), ' ended the snooze'];
case 'resolved': return person ? [strong(person), ' resolved the incident'] : ['Resolved: every alert stopped firing'];
case 'note': return [strong(person), ' added a note'];
case 'resolution_note': return [strong(person), ' noted what fixed it'];
case 'notified': {
const to = person ? strong(person) : 'the fallback topic';
if (ev.detail === 'reminder') return ['Reminder sent to ', to];
@@ -209,6 +220,22 @@ function eventText(ev) {
}
}
// Earlier incidents with the same signature that someone left notes on, the
// ones that recorded what fixed it first. Plain notes are on that incident's
// own page.
function similarSection() {
if (!similarList.length) return null;
return h('section', { class: 'section' },
h('h2', { class: 'section-title' }, h('span', { text: 'Seen before' })),
h('div', { class: 'card' },
h('ul', { class: 'similar' }, similarList.map((s) => h('li', { class: 'similar-item' },
h('a', { href: `/incidents/${s.id}`, text: `#${s.id} ${s.title}` }),
h('div', { class: 'sub', text: `${when(s.resolved_at)} · ${ago(s.resolved_at)}${s.note_count ? ` · ${s.note_count} note${s.note_count === 1 ? '' : 's'}` : ''}` }),
...s.resolution_notes.map((n) => h('div', { class: 'note note-fix', text: n.detail || '' })),
)))),
);
}
function timelineSection() {
const sorted = [...events].sort((a, b) => Date.parse(a.created_at) - Date.parse(b.created_at) || a.id - b.id);
return h('section', { class: 'section' },
@@ -223,20 +250,116 @@ function timelineSection() {
);
}
const isNote = (ev) => ev.type === 'note' || ev.type === 'resolution_note';
function timelineItem(ev) {
const mine = ev.type === 'note' && ev.user_id === myID();
const mine = isNote(ev) && ev.user_id === myID();
return h('li', { class: `tl-item tl-${ev.type}` },
h('span', { class: 'tl-dot' }),
h('div', { class: 'tl-body' },
h('div', { class: 'tl-text' }, eventText(ev)),
h('div', { class: 'tl-time', title: ev.created_at, text: `${when(ev.created_at)} · ${ago(ev.created_at)}` }),
ev.type === 'note' && h('div', { class: 'note', text: ev.detail || '' }),
isNote(ev) && h('div', { class: ev.type === 'resolution_note' ? 'note note-fix' : 'note', text: ev.detail || '' }),
mine && h('div', { class: 'note-actions' },
h('button', { class: 'btn btn-ghost btn-sm', type: 'button', onclick: () => deleteNote(ev) }, icon('trash'), 'Delete')),
),
);
}
// ---------- copy ----------
const fence = (rows) => ['```', ...rows, '```'];
const pairs = (obj) => Object.entries(obj || {}).sort(([a], [b]) => a.localeCompare(b)).map(([k, v]) => `${k}=${v}`);
// incidentMarkdown is everything on this page as text that reads well in a chat
// or an agent prompt. Times are ISO 8601, since "3 min ago" means nothing once
// it has been pasted somewhere else.
function incidentMarkdown() {
const out = [`# Incident #${inc.id}: ${inc.title}`, ''];
const add = (k, v) => { if (v != null && v !== '') out.push(`- ${k}: ${v}`); };
add('Status', inc.status);
add('Severity', inc.severity);
add('Team', inc.team_name);
add('Assigned to', inc.assigned_to_id != null ? inc.assigned_to || 'someone' : 'unassigned');
add('Triggered', inc.triggered_at);
if (inc.acknowledged_at) add('Acknowledged', `${inc.acknowledged_at} by ${inc.acknowledged_by || 'someone'}`);
if (isOpen() && isFuture(inc.snoozed_until)) add('Snoozed until', inc.snoozed_until);
if (inc.escalation_level > 0) add('Escalation level', inc.escalation_level);
if (inc.resolved_at) add('Resolved', `${inc.resolved_at} (${inc.resolution_source === 'manual' ? 'manually' : 'all alerts stopped firing'})`);
if (inc.archived_at) add('Archived', inc.archived_at);
const group = pairs(inc.group_labels);
if (group.length) out.push('- Grouped by:', ...group.map((g) => ` - ${g}`));
const alerts = inc.alerts || [];
out.push('', `## Alerts (${alerts.length})`);
for (const a of alerts) {
out.push('', `### ${a.name} (${a.status})`);
out.push(`- Started: ${a.starts_at}`);
if (a.status === 'resolved' && a.ends_at) out.push(`- Ended: ${a.ends_at}`);
if (a.generator_url) out.push(`- Source: ${a.generator_url}`);
const labels = pairs(a.labels);
if (labels.length) out.push('', 'Labels:', ...fence(labels));
const annotations = Object.entries(a.annotations || {}).sort(([x], [y]) => x.localeCompare(y));
if (annotations.length) out.push('', 'Annotations:', ...fence(annotations.map(([k, v]) => `${k}: ${v}`)));
}
const sorted = [...events].sort((a, b) => Date.parse(a.created_at) - Date.parse(b.created_at) || a.id - b.id);
if (sorted.length) {
out.push('', '## Timeline', '');
for (const ev of sorted) {
const text = eventText(ev, true).map((f) => (f instanceof Node ? f.textContent : f)).join('');
out.push(`- ${ev.created_at} ${text}`);
if (isNote(ev) && ev.detail) {
const label = ev.type === 'resolution_note' ? ' (what fixed it)' : '';
out.push(...(label ? [label] : []), ...ev.detail.split('\n').map((l) => ` > ${l}`));
}
}
}
if (similarList.length) {
out.push('', '## Seen before', '', 'Earlier incidents with the same signature:');
for (const s of similarList) {
out.push(`- #${s.id} ${s.title} (resolved ${s.resolved_at})`);
for (const n of s.resolution_notes || []) {
out.push(' - What fixed it:', ...(n.detail || '').split('\n').map((l) => ` > ${l}`));
}
}
}
out.push('', `_Copied from Terminal Duty at ${new Date().toISOString()}_`, '');
return out.join('\n');
}
// writeClipboard falls back to execCommand: the async API needs a secure
// context, and this server is often reached over plain HTTP.
async function writeClipboard(text) {
try {
await navigator.clipboard.writeText(text);
return;
} catch {
// fall through
}
const ta = h('textarea', { readonly: true, 'aria-hidden': 'true', class: 'clip-buffer' });
ta.value = text;
document.body.append(ta);
ta.select();
try {
if (!document.execCommand('copy')) throw new Error('copy refused');
} finally {
ta.remove();
}
}
async function copyIncident() {
if (!inc) return;
try {
await writeClipboard(incidentMarkdown());
toast('Copied incident');
} catch {
toast('Could not copy', 'error');
}
}
// ---------- actions ----------
const isOpen = () => inc.status !== 'resolved';
@@ -296,15 +419,35 @@ function unacknowledge() {
async function resolve() {
const id = inc.id;
const ok = await confirm({
title: 'Resolve this incident?',
const res = await openSheet(() => {
const textarea = h('textarea', {
name: 'resolution', autofocus: true, maxlength: '10000',
placeholder: 'What fixed it? Optional, shown on the next similar incident.',
});
const form = h('form', {
class: 'sheet-form',
onsubmit: (e) => {
e.preventDefault();
closeSheet({ resolution: textarea.value.trim() });
},
},
h('h2', { class: 'sheet-title', text: 'Resolve this incident?' }),
h('p', {
text: 'Resolving is final. If these alerts fire again they open a new incident, '
+ 'and if any are still firing this one stays closed regardless. '
+ 'Use snooze if you only need it out of the way.',
confirmLabel: 'Resolve',
danger: true,
}),
textarea,
h('div', { class: 'sheet-actions' },
h('button', { class: 'btn', type: 'button', onclick: () => closeSheet(null), text: 'Cancel' }),
h('button', { class: 'btn btn-danger', type: 'submit', text: 'Resolve' })),
);
textarea.addEventListener('keydown', (e) => {
if (e.key === 'Enter' && (e.ctrlKey || e.metaKey)) form.requestSubmit();
});
if (ok) await run(() => api.resolve(id), 'Resolved');
return form;
});
if (res) await run(() => api.resolve(id, res.resolution), 'Resolved');
}
function archive() {
@@ -382,16 +525,18 @@ async function addNote() {
const textarea = h('textarea', {
name: 'content', required: true, autofocus: true, placeholder: 'What did you find? What did you do?', maxlength: '10000',
});
const fix = h('input', { type: 'checkbox', name: 'fix' });
const form = h('form', {
class: 'sheet-form',
onsubmit: (e) => {
e.preventDefault();
const v = textarea.value.trim();
if (v) closeSheet(v);
if (v) closeSheet({ content: v, pinned: fix.checked });
},
},
h('h2', { class: 'sheet-title', text: 'Add note' }),
textarea,
h('label', { class: 'check' }, fix, ' This is what fixed it (shown on similar incidents)'),
h('div', { class: 'sheet-actions' },
h('button', { class: 'btn', type: 'button', onclick: () => closeSheet(null), text: 'Cancel' }),
h('button', { class: 'btn btn-primary', type: 'submit', text: 'Save note' })),
@@ -402,7 +547,7 @@ async function addNote() {
});
return form;
});
if (content) await run(() => api.addNote(id, content), 'Note added');
if (content) await run(() => api.addNote(id, content.content, content.pinned), 'Note added');
}
async function deleteNote(ev) {
@@ -422,10 +567,12 @@ async function moreMenu() {
items.push(item('user', 'Assign…', assign));
items.push(isSnoozed() ? item('bell', 'End snooze', unsnooze) : item('clock', 'Snooze…', snooze));
items.push(item('note', 'Add note…', addNote));
items.push(item('copy', 'Copy incident', copyIncident));
items.push(h('li', { class: 'menu-sep', role: 'separator' }));
items.push(item('checkCircle', 'Resolve…', resolve, 'danger'));
} else {
items.push(item('note', 'Add note…', addNote));
items.push(item('copy', 'Copy incident', copyIncident));
items.push(inc.archived_at ? item('undo', 'Unarchive', unarchive) : item('archive', 'Archive', archive));
}
@@ -454,6 +601,7 @@ export function key(e) {
case 'z': if (isOpen() && !isSnoozed()) snooze(); return true;
case 'Z': if (isSnoozed()) unsnooze(); return true;
case 'c': addNote(); return true;
case 'y': copyIncident(); return true;
case 'x': if (!isOpen()) (inc.archived_at ? unarchive() : archive()); return true;
default: return false;
}
+168
View File
@@ -0,0 +1,168 @@
// Statistics: how many incidents, how fast they are answered, and when and
// what the alerts are. The same figures the TUI's Stats tab shows, over a
// range picked with the chips. The server scopes them to the caller's teams.
import * as api from './api.js';
import { h, clear, emptyState, spinner } from './ui.js';
import { duration } from './format.js';
const DAY_MS = 24 * 60 * 60 * 1000;
// `days` counts back from today, inclusive; the server reads from/to as UTC
// dates, so these are too.
const RANGES = [
{ id: 'today', label: 'Today', days: 1 },
{ id: '7d', label: '7d', days: 7 },
{ id: '30d', label: '30d', days: 30 },
{ id: '90d', label: '90d', days: 90 },
{ id: 'all', label: 'All', days: null },
];
const view = () => document.getElementById('view-stats');
let range = '30d';
let data = null;
let error = null;
const utcDate = (ms) => new Date(ms).toISOString().slice(0, 10);
function query(r) {
if (!r.days) return {};
const now = Date.now();
return { from: utcDate(now - (r.days - 1) * DAY_MS), to: utcDate(now) };
}
export function show() {
render();
refresh();
}
export async function refresh() {
const requested = range;
const q = query(RANGES.find((x) => x.id === range));
try {
const [incidents, top, byHour, byDay] = await Promise.all([
api.statsIncidents(q),
api.statsTop({ ...q, limit: 10 }),
api.statsByHour(q),
api.statsByDay(q),
]);
if (requested !== range) return;
data = { incidents, top, byHour, byDay };
error = null;
} catch (err) {
if (requested !== range) return;
error = err.message;
}
render();
}
function setRange(id) {
if (id === range) return;
range = id;
data = null;
render();
refresh();
}
function render() {
const chips = h('div', { class: 'chips', role: 'tablist', 'aria-label': 'Time range' },
RANGES.map((r) => h('button', {
class: 'chip',
type: 'button',
role: 'tab',
'aria-selected': String(r.id === range),
onclick: () => setRange(r.id),
text: r.label,
})));
let body;
if (error && !data) body = h('div', { class: 'load-error', text: error });
else if (!data) body = spinner();
else if (!data.incidents.total && !data.byHour.some((x) => x.count)) {
body = emptyState('No data in this range', '', 'chart');
} else {
body = h('div', { class: 'stats' },
error && h('div', { class: 'load-error', text: `Showing older data: ${error}` }),
tiles(data.incidents),
data.top.length > 0 && card('Top alerts', topAlerts(data.top)),
card('Alerts by hour (UTC)', columns(
data.byHour.map((x) => ({ label: String(x.hour), value: x.count, tick: x.hour % 6 === 0 })),
'Alerts by hour of day')),
card('Alerts by day', columns(
data.byDay.map((x) => ({ label: x.day_name.slice(0, 3), value: x.count, tick: true })),
'Alerts by day of week')));
}
clear(view(), h('div', {}, chips, body));
}
// A missing mean means nothing has been acknowledged or resolved yet.
const mean = (s) => (s == null ? '—' : duration(s * 1000));
function tiles(s) {
const tile = (label, value, cls = '') => h('div', { class: `stat-tile ${cls}` },
h('div', { class: 'stat-value', text: String(value) }),
h('div', { class: 'stat-label', text: label }));
return h('div', { class: 'stat-tiles' },
tile('Incidents', s.total),
tile('Triggered', s.triggered, 'st-triggered'),
tile('Acknowledged', s.acknowledged, 'st-acknowledged'),
tile('Resolved', s.resolved, 'st-resolved'),
tile('Mean time to acknowledge', mean(s.mtta_seconds)),
tile('Mean time to resolve', mean(s.mttr_seconds)));
}
function card(title, content) {
return h('section', { class: 'chart-card card card-pad' },
h('h3', { class: 'chart-title', text: title }), content);
}
// Ranked names with a bar scaled to the busiest one.
function topAlerts(items) {
const max = Math.max(...items.map((x) => x.count), 1);
return h('ol', { class: 'hbars' }, items.map((x) => {
const fill = h('span', { class: 'hbar-fill' });
fill.style.width = `${Math.max(2, (x.count / max) * 100)}%`;
return h('li', { class: 'hbar' },
h('span', { class: 'hbar-name', title: x.name, text: x.name }),
h('span', { class: 'hbar-track' }, fill),
h('span', { class: 'hbar-count', text: String(x.count) }));
}));
}
const SVG_NS = 'http://www.w3.org/2000/svg';
function svg(tag, attrs = {}, text) {
const el = document.createElementNS(SVG_NS, tag);
for (const [k, v] of Object.entries(attrs)) el.setAttribute(k, String(v));
if (text != null) el.textContent = text;
return el;
}
// A column chart: one bar per item, the value in a tooltip, and a label under
// the items marked `tick`.
function columns(items, label) {
const W = 480;
const H = 140;
const base = H - 18;
const step = W / items.length;
const max = Math.max(...items.map((x) => x.value), 1);
const root = svg('svg', {
class: 'columns', viewBox: `0 0 ${W} ${H}`, role: 'img', 'aria-label': label,
});
root.appendChild(svg('line', { class: 'axis', x1: 0, x2: W, y1: base, y2: base }));
items.forEach((it, i) => {
const bh = it.value ? Math.max(2, (it.value / max) * (base - 6)) : 0;
const x = i * step + step * 0.15;
const g = svg('g', { class: 'col' });
g.appendChild(svg('title', {}, `${it.label}: ${it.value}`));
// A full-height transparent hit area, so a tiny bar is still hoverable.
g.appendChild(svg('rect', { class: 'col-hit', x: i * step, y: 0, width: step, height: base }));
if (bh) g.appendChild(svg('rect', { class: 'col-bar', x, y: base - bh, width: step * 0.7, height: bh, rx: 2 }));
root.appendChild(g);
if (it.tick) {
root.appendChild(svg('text', { class: 'col-label', x: i * step + step / 2, y: H - 4, 'text-anchor': 'middle' }, it.label));
}
});
return root;
}
+568 -136
View File
@@ -18,9 +18,9 @@
// than no form, but it is not the thing enforcing anything.
import * as api from './api.js';
import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard } from './ui.js';
import { h, clear, spinner, confirm, icon, openSheet, closeSheet, menuCard, badge, labelChip } from './ui.js';
import { state, currentTeam, users as allUsers, myID } from './state.js';
import { isoDate, addDays, mondayOf, initial } from './format.js';
import { isoDate, addDays, mondayOf, isoWeek, initial, ago, when, duration } from './format.js';
const view = () => document.getElementById('view-team');
@@ -52,12 +52,10 @@ let freshKey = null; // an integration key, shown once, until the view is left
export function show(route) {
const next = route?.tab ?? null;
// A different sub-section wants different data, so the old answer goes
// rather than being shown under the new heading until the fetch lands. The
// ladder draft goes with it: it is an edit of the page being left.
// rather than being shown under the new heading until the fetch lands.
if (next !== tab) {
tab = next;
data = null;
draft = null;
}
if (!data) clear(view(), subnav(), spinner());
refresh();
@@ -111,14 +109,14 @@ async function load(id) {
return { members, escalation };
}
if (tab === 'sources') return { integrations: await api.integrations(id) };
if (tab === 'deadman') return { deadman: await api.deadman(id) };
if (tab === 'deadman') return { deadman: await api.deadmanSwitches(id) };
const grid = gridDays();
const [members, integrations, escalation, deadman, schedule] = await Promise.all([
api.teamMembers(id),
api.integrations(id),
api.escalation(id),
api.deadman(id),
api.deadmanSwitches(id),
api.schedule(id, isoDate(grid.start), isoDate(addDays(grid.start, grid.count - 1))),
]);
return { members, integrations, escalation, deadman, schedule };
@@ -185,7 +183,6 @@ function teamPicker() {
teamID = Number(select.value);
data = null;
freshKey = null;
draft = null;
refresh();
});
return h('div', { class: 'card' }, h('h2', { text: 'Team' }), select);
@@ -203,8 +200,8 @@ function overview() {
const levels = (data.escalation?.levels || []).length;
const keys = (data.integrations || []).length;
const unused = (data.integrations || []).filter((i) => !i.last_used_at).length;
const switches = (data.deadman?.matchers || '')
.split(';').map((x) => x.trim()).filter(Boolean).length;
const switches = (data.deadman || []).length;
const dead = (data.deadman || []).filter((s) => s.status === 'dead').length;
return h('div', { class: 'overview-menu' },
menuCard('/team/rota', 'Rota', null,
@@ -220,7 +217,9 @@ function overview() {
? (unused ? `${unused} of them never used.` : 'All in use.')
: 'No key yet, so nothing can reach this team.'),
menuCard('/team/deadman', 'Dead man’s switches', switches || null,
switches ? 'Alerts whose absence opens an incident.' : 'Nothing watched.'),
switches
? (dead ? `${dead} of them silent.` : 'All quiet, as they should be.')
: 'Nothing watched.'),
);
}
@@ -241,6 +240,7 @@ function overview() {
const monthFmt = new Intl.DateTimeFormat(undefined, { month: 'long', year: 'numeric' });
const weekdayFmt = new Intl.DateTimeFormat(undefined, { weekday: 'short' });
const dayShortFmt = new Intl.DateTimeFormat(undefined, { day: 'numeric', month: 'short' });
const longDayFmt = new Intl.DateTimeFormat(undefined, {
weekday: 'long', day: 'numeric', month: 'long',
});
@@ -282,11 +282,14 @@ function scheduleCard() {
const key = isoDate(d);
const e = byDate.get(key);
const inMonth = d.getMonth() === month;
// Every row starts with its week number, which is also the way to fill the
// whole week at once.
if (i % 7 === 0) cells.push(weekCell(d, byDate, today));
if (inMonth && e && !seen.has(e.user_id)) seen.set(e.user_id, e.username);
cells.push(dayCell(d, key, e, inMonth, today));
}
const heads = [];
const heads = [h('span', { class: 'rota-wd', title: 'ISO week number', text: 'Wk' })];
for (let i = 0; i < 7; i++) {
// Any Monday will do; this one is a Monday.
heads.push(h('span', { class: 'rota-wd', text: weekdayFmt.format(new Date(2024, 0, 1 + i)) }));
@@ -324,6 +327,23 @@ function scheduleCard() {
];
}
// The ISO week number at the start of a row. For an owner it is a button: one
// tap fills the week, which is the way a rota is usually handed out — a person
// takes a week, not seven separate days.
function weekCell(monday, byDate, today) {
const n = isoWeek(monday);
const current = isoDate(monday) <= today && today < isoDate(addDays(monday, 7));
const cls = `rota-week${current ? ' current' : ''}`;
const label = `Week ${n}`;
return isOwner()
? h('button', {
class: cls, type: 'button', text: String(n),
title: `${label} · assign somebody for the whole week`, 'aria-label': label,
onclick: () => weekSheet(monday, byDate),
})
: h('div', { class: cls, title: label, text: String(n) });
}
function dayCell(d, key, e, inMonth, today) {
const cls = ['rota-day', !inMonth && 'outside', key === today && 'today', key < today && 'past']
.filter(Boolean).join(' ');
@@ -379,6 +399,73 @@ function coverNote(byDate) {
' left this month with nobody on call.');
}
// One week, in the sheet: who holds each day of it, and one person to put on
// all of them. Days already gone are left alone — who was on call last Tuesday
// is a fact, and "the whole week" should not rewrite it — and the week's
// overhang into the next month is included, since it is the same week.
function weekSheet(monday, byDate) {
const today = isoDate(new Date());
const days = Array.from({ length: 7 }, (_, i) => addDays(monday, i));
const keys = days.map(isoDate);
const ahead = keys.filter((k) => k >= today);
const range = `${dayShortFmt.format(days[0])} – ${dayShortFmt.format(days[6])}`;
const who = memberSelect();
const onlyEmpty = h('input', { type: 'checkbox' });
const problem = h('p', { class: 'load-error', hidden: true });
const holders = h('div', { class: 'week-holders' }, days.map((d, i) => {
const e = byDate.get(keys[i]);
return h('span', {
class: `week-holder${keys[i] < today ? ' past' : ''}`,
title: `${keys[i]} · ${e ? e.username : 'nobody'}`,
},
h('span', { class: 'rota-num', text: weekdayFmt.format(d) }),
e
? h('span', { class: `rota-chip ${colorClass(e.user_id)}`, text: initial(e.username) })
: h('span', { class: 'rota-chip none' }));
}));
openSheet(() => [
h('h2', { class: 'sheet-title', text: `Week ${isoWeek(monday)}` }),
h('p', { class: 'sheet-text', text: range }),
holders,
ahead.length
? [
h('label', { class: 'sheet-pick' }, 'On call ', who),
h('label', { class: 'checkbox' }, onlyEmpty, ' Only fill days nobody has yet'),
h('p', { class: 'muted small' },
ahead.length < 7
? `Days already past are left alone, so this covers the ${ahead.length} still to come. `
: '',
'Anybody already on those days is replaced unless you tick the box.'),
]
: h('p', { class: 'muted', text: 'This whole week is already over.' }),
problem,
h('div', { class: 'sheet-actions' },
h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet() }),
ahead.length > 0 && h('button', {
class: 'btn btn-primary', type: 'button', autofocus: true, text: 'Assign week',
onclick: () => {
const dates = onlyEmpty.checked ? ahead.filter((k) => !byDate.has(k)) : ahead;
if (!who.value) {
problem.textContent = 'There is nobody in this team to assign.';
problem.hidden = false;
return;
}
if (!dates.length) {
problem.textContent = 'Every day still to come already has somebody.';
problem.hidden = false;
return;
}
closeSheet();
act(() => api.assignSchedule(teamID, Number(who.value), dates, !onlyEmpty.checked));
},
}),
),
]);
}
// One day, in the sheet: who has it, who should, and the way to empty it. This
// is where the per-row Clear button went — the grid has no room for thirty of
// them, and the day you want to change is the one you just tapped.
@@ -455,15 +542,84 @@ function memberSelect(selected) {
// --- escalation ------------------------------------------------------------
// The ladder is edited as a whole and sent as a whole, because the API replaces
// it wholesale: the levels are an order, and patching one rung would leave the
// numbering of the others undecided.
let draft = null;
const LEVEL_STATUS = {
ready: { label: 'Ready', hint: 'Somebody here can be woken.' },
escalating: { label: 'Escalating', hint: 'An unanswered incident has climbed to this level.' },
unreachable: { label: 'Pages nobody', hint: 'Nobody on this level can be woken right now.' },
};
const levelBadge = (status) => statusBadge(LEVEL_STATUS, status, 'ready');
// One target as the list shows it: who it means today, and why it would not
// wake them if it would not.
function targetLine(t) {
const label = t.kind === 'oncall'
? `On call${t.username ? ` · ${t.username}` : ''}`
: (t.username || 'Unknown person');
return h('div', { class: 'target-line' },
h('span', { text: label }),
t.problem && h('span', { class: 'target-problem', text: t.problem }));
}
function escalationCard() {
const esc = data.escalation;
if (!draft) {
draft = {
const esc = data.escalation || {};
const levels = esc.levels || [];
const rows = levels.map((l) => h('tr', {},
h('td', {}, h('strong', { text: `Level ${l.position}` })),
h('td', {}, levelBadge(l.status)),
h('td', { class: 'wrap' }, ...l.targets.map(targetLine)),
h('td', { class: 'muted small', text: duration(l.timeout_seconds * 1000) }),
h('td', { class: 'small' }, l.waiting?.length
? l.waiting.flatMap((id, i) => [i > 0 && ', ', h('a', { href: `/incidents/${id}`, text: `#${id}` })])
: h('span', { class: 'muted', text: '—' })),
));
const facts = [];
if (levels.length) {
const n = esc.repeat_count || 0;
if (n) facts.push(`Then the whole ladder repeats ${n} more ${n === 1 ? 'time' : 'times'}.`);
facts.push(esc.fallback_topic
? ['Finally the ntfy topic ', h('code', { text: esc.fallback_topic }), ' is paged once.']
: 'No fallback topic: after the last level the chain just ends.');
facts.push(esc.last_escalated_at
? ['Last escalated ',
h('span', { title: when(esc.last_escalated_at), text: ago(esc.last_escalated_at) }),
' on ', h('a', { href: `/incidents/${esc.last_escalated_incident_id}`, text: `#${esc.last_escalated_incident_id}` }), '.']
: 'Nothing has needed to escalate yet.');
}
return h('div', { class: 'card' },
h('div', { class: 'card-head' },
h('h2', { text: 'Escalation' }),
isOwner() && h('button', {
class: 'btn', type: 'button', onclick: openLadderEditor,
text: levels.length ? 'Edit ladder' : 'Set up ladder',
})),
h('p', { class: 'muted small' },
'When a level’s wait passes and nobody has acknowledged, the next level is ',
'paged. Acknowledging or resolving stops it; snoozing pauses it.'),
levels.length
? h('div', { class: 'table-scroll' },
h('table', { class: 'admin-table status-table' },
h('thead', {}, h('tr', {},
h('th', { text: 'Level' }), h('th', { text: 'Status' }), h('th', { text: 'Pages' }),
h('th', { text: 'Then after' }), h('th', { text: 'Waiting now' }))),
h('tbody', {}, rows)))
: h('p', { class: 'muted' },
'No ladder. An unacknowledged incident re-pages the same person every ',
'reminder interval and nobody else is woken.'),
...facts.map((f) => h('p', { class: 'muted small' }, f)),
);
}
// The ladder is edited as a whole and sent as a whole, because the API replaces
// it wholesale: the levels are an order, and patching one rung would leave the
// numbering of the others undecided. The draft lives in the sheet, so a poll of
// the page underneath cannot throw away half an edit.
function openLadderEditor() {
const esc = data.escalation || {};
const draft = {
repeat_count: esc.repeat_count || 0,
fallback_topic: esc.fallback_topic || '',
levels: (esc.levels || []).map((l) => ({
@@ -471,41 +627,39 @@ function escalationCard() {
targets: (l.targets || []).map((t) => ({ kind: t.kind, user_id: t.user_id })),
})),
};
}
const body = [];
const body = h('div', { class: 'ladder-editor' });
const problem = h('p', { class: 'load-error', hidden: true });
const paint = () => {
const parts = [];
if (!draft.levels.length) {
body.push(h('p', { class: 'muted' },
'No ladder. An unacknowledged incident re-pages the same person every ',
'reminder interval and nobody else is woken.'));
parts.push(h('p', { class: 'muted small' }, 'No levels yet. Add the first one.'));
}
draft.levels.forEach((level, i) => {
body.push(h('div', { class: 'ladder-level' },
parts.push(h('div', { class: 'ladder-level' },
h('div', { class: 'ladder-head' },
h('strong', { text: `Level ${i + 1}` }),
isOwner() && h('button', {
h('button', {
class: 'btn-sm danger', type: 'button', text: 'Remove',
onclick: () => { draft.levels.splice(i, 1); render(); },
onclick: () => { draft.levels.splice(i, 1); paint(); },
})),
h('label', {}, 'Wait ', minutesInput(level.timeout_seconds, (secs) => {
level.timeout_seconds = secs;
}), ' before the next level'),
h('div', { class: 'ladder-targets' },
...level.targets.map((t, ti) => targetRow(level, t, ti)),
isOwner() && h('button', {
...level.targets.map((t, ti) => targetRow(level, t, ti, paint)),
h('button', {
class: 'btn-sm', type: 'button', text: '+ target',
onclick: () => { level.targets.push({ kind: 'oncall' }); render(); },
onclick: () => { level.targets.push({ kind: 'oncall' }); paint(); },
})),
));
});
if (isOwner()) {
body.push(h('button', {
parts.push(h('button', {
class: 'btn-sm', type: 'button', text: '+ level',
onclick: () => {
draft.levels.push({ timeout_seconds: 300, targets: [{ kind: 'oncall' }] });
render();
paint();
},
}));
@@ -518,31 +672,43 @@ function escalationCard() {
type: 'text', value: draft.fallback_topic, placeholder: 'terdut-oncall-all',
oninput: (e) => { draft.fallback_topic = e.target.value; },
});
body.push(h('label', {}, 'Repeat the whole ladder ', repeat, ' more times'));
body.push(h('label', {}, 'Then page this ntfy topic once ', fallback));
body.push(h('button', {
class: 'btn', type: 'button', text: 'Save ladder',
onclick: () => act(() => api.setEscalation(teamID, draft), { resetDraft: true }),
}));
parts.push(h('label', {}, 'Repeat the whole ladder ', repeat, ' more times'));
parts.push(h('label', {}, 'Then page this ntfy topic once ', fallback));
clear(body, ...parts);
};
paint();
const save = h('button', { class: 'btn btn-primary', type: 'button', text: 'Save ladder' });
save.addEventListener('click', async () => {
try {
await api.setEscalation(teamID, draft);
} catch (err) {
problem.textContent = err.message;
problem.hidden = false;
return;
}
closeSheet(true);
refresh();
});
openSheet(() => [
h('h2', { class: 'sheet-title', text: 'Edit ladder' }),
body,
problem,
h('div', { class: 'sheet-actions' },
h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }),
save),
]);
}
return h('div', { class: 'card' },
h('h2', { text: 'Escalation' }),
h('p', { class: 'muted small' },
'When a level’s wait passes and nobody has acknowledged, the next level is ',
'paged. Acknowledging or resolving stops it; snoozing pauses it.'),
...body,
);
}
function targetRow(level, target, index) {
function targetRow(level, target, index, repaint) {
const kind = h('select', {},
h('option', { value: 'oncall', text: 'Whoever is on call', selected: target.kind === 'oncall' }),
h('option', { value: 'user', text: 'A specific person', selected: target.kind === 'user' }));
kind.addEventListener('change', () => {
target.kind = kind.value;
target.user_id = kind.value === 'user' ? (data.members[0] || {}).user_id : undefined;
render();
repaint();
});
const who = target.kind === 'user'
@@ -553,10 +719,10 @@ function targetRow(level, target, index) {
}
return h('div', { class: 'target-row' }, kind, who,
isOwner() && h('button', {
h('button', {
class: 'btn-sm danger', type: 'button', text: '×',
title: 'Remove this target',
onclick: () => { level.targets.splice(index, 1); render(); },
onclick: () => { level.targets.splice(index, 1); repaint(); },
}));
}
@@ -574,33 +740,54 @@ function minutesInput(seconds, onChange) {
function integrationsCard() {
const rows = (data.integrations || []).map((i) =>
h('tr', {},
h('td', {}, h('strong', { text: i.name })),
h('td', { class: 'muted small', text: i.kind }),
h('td', { class: 'muted small', text: i.last_used_at ? 'in use' : 'never used' }),
h('td', {}, isOwner() && h('button', {
h('td', {}, sourceBadge(i.status)),
h('td', { class: 'wrap' },
h('strong', { text: i.name }),
h('div', { class: 'muted small', text: i.kind })),
// When the key last posted, and when an alert last arrived on it. They
// differ: a payload with nothing usable in it stamps only the first.
h('td', { class: 'muted small' }, timeCell(i.last_used_at)),
h('td', { class: 'muted small' }, timeCell(i.last_alert_at)),
h('td', { class: 'muted small num', title: 'Distinct alerts refreshed in the last 24 hours',
text: String(i.alerts_24h ?? 0) }),
h('td', { class: 'muted small' }, h('span', { title: when(i.created_at), text: ago(i.created_at) })),
h('td', {}, isOwner() && h('div', { class: 'row-actions' },
h('button', {
class: 'btn-sm', type: 'button', text: 'Rename', onclick: () => openRenameSource(i),
}),
h('button', {
class: 'btn-sm danger', type: 'button', text: 'Revoke',
onclick: async () => {
if (!(await confirm({
title: `Revoke ${i.name}?`,
text: 'Anything posting with this key stops delivering immediately.',
text: 'Anything posting with this key stops delivering immediately. Alerts it already delivered stay.',
confirmLabel: 'Revoke',
danger: true,
}))) return;
act(() => api.deleteIntegration(teamID, i.id));
},
})),
}))),
));
return h('div', { class: 'card' },
h('div', { class: 'card-head' },
h('h2', { text: 'Alert sources' }),
isOwner() && h('button', {
class: 'btn', type: 'button', text: 'New source', onclick: openNewSource,
})),
h('p', { class: 'muted small' },
'Alerts arrive on an integration key, which says both that the sender may ',
'post and which team the alerts belong to.'),
rows.length
? h('table', { class: 'admin-table' }, h('tbody', {}, rows))
: h('p', { class: 'muted', text: 'No alert source yet, so nothing can reach this team.' }),
freshKey && newKeyPanel(),
isOwner() && !freshKey && newIntegrationForm(),
rows.length
? h('div', { class: 'table-scroll' },
h('table', { class: 'admin-table status-table' },
h('thead', {}, h('tr', {},
h('th', { text: 'Status' }), h('th', { text: 'Source' }),
h('th', { text: 'Last webhook' }), h('th', { text: 'Last alert' }),
h('th', { class: 'num', text: 'Alerts 24h' }), h('th', { text: 'Created' }), h('th'))),
h('tbody', {}, rows)))
: h('p', { class: 'muted', text: 'No alert source yet, so nothing can reach this team.' }),
);
}
@@ -631,103 +818,350 @@ function newKeyPanel() {
);
}
function newIntegrationForm() {
const name = h('input', { type: 'text', placeholder: 'prod alertmanager', required: true });
const form = h('form', { class: 'inline-form' }, name,
h('button', { class: 'btn', type: 'submit', text: 'Add' }));
// A sheet with one name field, for adding a source and for renaming one: the two
// differ only in what they call and what they put in the box.
function openNameSheet({ title, submit, value, run }) {
const name = h('input', {
type: 'text', placeholder: 'prod alertmanager', required: true, value, autofocus: true,
});
const problem = h('p', { class: 'load-error', hidden: true });
const form = h('form', { class: 'stacked-form' },
h('label', {}, 'Name ', name),
problem,
h('div', { class: 'sheet-actions' },
h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }),
h('button', { class: 'btn btn-primary', type: 'submit', text: submit })));
form.addEventListener('submit', async (e) => {
e.preventDefault();
try {
freshKey = await api.createIntegration(teamID, name.value.trim());
await refresh();
await run(name.value.trim());
} catch (err) {
error = err.message;
render();
problem.textContent = err.message;
problem.hidden = false;
return;
}
closeSheet(true);
refresh();
});
openSheet(() => [h('h2', { class: 'sheet-title', text: title }), form]);
}
function openNewSource() {
openNameSheet({
title: 'New source', submit: 'Add source', value: '',
// The key comes back once, and the card shows it until dismissed.
run: async (name) => { freshKey = await api.createIntegration(teamID, name); },
});
}
function openRenameSource(i) {
openNameSheet({
title: `Rename ${i.name}`, submit: 'Rename', value: i.name,
run: (name) => api.renameIntegration(teamID, i.id, name),
});
return form;
}
// --- dead man's switches ---------------------------------------------------
// Status badges, shared by the Sources and Switches lists: a table of label and
// hint per status, and one function to draw it. Module-level, so the two cards
// can be defined in either order.
const SWITCH_STATUS = {
healthy: { label: 'Healthy', hint: 'Heard from within its timeout.' },
dead: { label: 'Dead', hint: 'Silent for longer than its timeout.' },
dormant: { label: 'Dormant', hint: 'Nothing has matched yet, so there is nothing to lose.' },
};
const SOURCE_STATUS = {
active: { label: 'Active', hint: 'Posted within the last day.' },
quiet: { label: 'Quiet', hint: 'Has posted, but not in the last day. Nothing firing is a fine reason.' },
never: { label: 'Never used', hint: 'Nothing has been posted with this key yet.' },
};
function statusBadge(table, status, fallback) {
const s = table[status] || table[fallback];
const el = badge(s.label, `st-${status}`);
el.title = s.hint;
return el;
}
const switchBadge = (status) => statusBadge(SWITCH_STATUS, status, 'dormant');
const sourceBadge = (status) => statusBadge(SOURCE_STATUS, status, 'never');
const timeCell = (iso) => iso
? h('span', { title: when(iso), text: ago(iso) })
: h('span', { class: 'muted', text: 'never' });
// When it last opened an incident. An incident that is still open is a link,
// because that is the thing somebody looking at a red row wants next.
const triggeredCell = (iso, incidentID) => {
if (!iso) return h('span', { class: 'muted', text: 'never' });
return incidentID
? h('a', { href: `/incidents/${incidentID}`, title: when(iso) }, `#${incidentID} · ${ago(iso)}`)
: h('span', { title: when(iso), text: ago(iso) });
};
function switchRows(sw) {
const main = h('tr', {},
h('td', {}, switchBadge(sw.status)),
h('td', { class: 'wrap' },
h('strong', { text: sw.name }),
sw.name !== sw.matcher && h('div', { class: 'muted small' }, h('code', { text: sw.matcher }))),
h('td', { class: 'muted small' }, timeCell(sw.last_heartbeat_at)),
h('td', { class: 'muted small' }, triggeredCell(sw.last_triggered_at, sw.open_incident_id)),
h('td', { class: 'muted small', text: duration(sw.timeout_seconds * 1000) }),
h('td', {}, isOwner() && h('button', {
class: 'btn-sm danger', type: 'button', text: 'Remove',
onclick: async () => {
if (!(await confirm({
title: `Remove ${sw.name}?`,
text: 'It stops being watched. An incident it already opened stays open until it is resolved.',
confirmLabel: 'Remove',
danger: true,
}))) return;
act(() => api.deleteDeadmanSwitch(teamID, sw.id));
},
})),
);
// One heartbeat is the switch's own times; several are worth telling apart,
// since a live cluster must not hide a dead one.
const sources = sw.sources.length > 1
? sw.sources.map((src) => h('tr', { class: 'source-row' },
h('td', {}, switchBadge(src.status)),
h('td', { class: 'source-labels' },
...Object.entries(src.labels || {})
.filter(([k]) => k !== 'alertname')
.map(([k, v]) => labelChip(k, v)),
!Object.keys(src.labels || {}).some((k) => k !== 'alertname')
&& h('code', { class: 'small', text: src.fingerprint })),
h('td', { class: 'muted small' }, timeCell(src.last_heartbeat_at)),
h('td', { class: 'muted small' }, triggeredCell(src.last_triggered_at, src.incident_id)),
h('td'), h('td')))
: [];
return [main, ...sources];
}
function deadmanCard() {
const d = data.deadman || {};
const matchers = h('input', {
type: 'text', value: d.matchers || '', placeholder: 'alertname=Watchdog',
class: 'wide',
});
const timeout = h('input', {
type: 'number', min: '0', class: 'setting-value',
value: String(Math.round((d.timeout_seconds || 0) / 60)),
});
const severity = h('select', {},
...['critical', 'error', 'warning', 'info'].map((s) =>
h('option', { value: s, text: s, selected: (d.severity || 'critical') === s })));
const form = h('form', { class: 'stacked-form' },
h('label', {}, 'Heartbeat alerts ', matchers),
h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'),
h('label', {}, 'Open the incident at severity ', severity),
h('button', { class: 'btn', type: 'submit', text: 'Save switches' }));
form.addEventListener('submit', (e) => {
e.preventDefault();
act(() => api.setDeadman(teamID, {
matchers: matchers.value.trim(),
timeout_seconds: Number(timeout.value) * 60,
severity: severity.value,
}));
});
const switches = data.deadman || [];
return h('div', { class: 'card' },
h('div', { class: 'card-head' },
h('h2', { text: 'Dead man’s switches' }),
isOwner() && h('button', {
class: 'btn', type: 'button', text: 'New switch', onclick: openNewSwitch,
})),
h('p', { class: 'muted small' },
'Alerts whose ABSENCE is the signal. Receiving one opens nothing; going ',
'quiet for longer than the timeout opens an incident. ',
h('code', { text: 'alertname=Watchdog,cluster=prod; alertname=EdgeHeartbeat' }),
' — semicolons separate switches, commas separate conditions, and every ',
'switch must name an alertname. Leave empty to watch nothing.'),
isOwner() ? form : h('p', { class: 'muted', text: d.matchers || 'Nothing watched.' }),
'quiet for longer than the switch’s timeout opens an incident.'),
switches.length
? h('div', { class: 'table-scroll' },
h('table', { class: 'admin-table status-table' },
h('thead', {}, h('tr', {},
h('th', { text: 'Status' }), h('th', { text: 'Switch' }),
h('th', { text: 'Last heartbeat' }), h('th', { text: 'Last triggered' }),
h('th', { text: 'Silent after' }), h('th'))),
h('tbody', {}, switches.flatMap(switchRows))))
: h('p', { class: 'muted', text: 'Nothing watched.' }),
);
}
// The form lives in the sheet, not on the page: most visits are to look at the
// list, and a form that is always open is the page this replaced.
function openNewSwitch() {
const name = h('input', { type: 'text', placeholder: 'Prod Watchdog', autofocus: true });
const matcher = h('input', {
type: 'text', placeholder: 'alertname=Watchdog,cluster=prod', class: 'wide', required: true,
});
const timeout = h('input', {
type: 'number', min: '1', value: '15', class: 'setting-value', required: true,
});
const severity = h('select', {},
...['critical', 'error', 'warning', 'info'].map((s) => h('option', { value: s, text: s })));
const problem = h('p', { class: 'load-error', hidden: true });
const form = h('form', { class: 'stacked-form' },
h('label', {}, 'Name (optional) ', name),
h('label', {}, 'Heartbeat alert ', matcher),
h('p', { class: 'muted small' },
'Conditions are ', h('code', { text: 'label=value' }), ' separated by commas, and one ',
'must be ', h('code', { text: 'alertname' }), '. Every distinct label set that ',
'matches is watched on its own.'),
h('label', {}, 'Declare dead after ', timeout, ' minutes of silence'),
h('label', {}, 'Open the incident at severity ', severity),
problem,
h('div', { class: 'sheet-actions' },
h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }),
h('button', { class: 'btn btn-primary', type: 'submit', text: 'Add switch' })));
form.addEventListener('submit', async (e) => {
e.preventDefault();
try {
await api.createDeadmanSwitch(teamID, {
name: name.value.trim(),
matcher: matcher.value.trim(),
timeout_seconds: Math.round(Number(timeout.value) * 60),
severity: severity.value,
});
} catch (err) {
problem.textContent = err.message;
problem.hidden = false;
return;
}
closeSheet(true);
refresh();
});
openSheet(() => [h('h2', { class: 'sheet-title', text: 'New switch' }), form]);
}
// --- members ---------------------------------------------------------------
function membersCard() {
const rows = (data.members || []).map((m) =>
h('tr', {},
h('td', {}, h('strong', { text: m.username })),
h('td', { class: 'muted small', text: m.role }),
h('td', {}, isOwner() && h('button', {
class: 'btn-sm', type: 'button',
text: m.role === 'owner' ? 'Make member' : 'Make owner',
onclick: () => act(() =>
api.addTeamMember(teamID, m.user_id, m.role === 'owner' ? 'member' : 'owner')),
}), isOwner() && h('button', {
class: 'btn-sm danger', type: 'button', text: 'Remove',
onclick: () => act(() => api.removeTeamMember(teamID, m.user_id)),
})),
));
const MEMBER_STATUS = {
oncall: { label: 'On call', hint: 'The rota has them today.' },
reachable: { label: 'Reachable', hint: 'Has an ntfy topic, so a page would reach them.' },
unpageable: { label: 'Can’t be paged', hint: 'A page to them would go nowhere.' },
};
const memberBadge = (m) => {
const el = statusBadge(MEMBER_STATUS, m.status, 'reachable');
if (m.problem) el.title = `${MEMBER_STATUS.unpageable.hint} ${m.problem}.`;
return el;
};
// Rota days are UTC dates with no time in them; formatting one in the viewer's
// zone could show the day before.
const shiftFmt = new Intl.DateTimeFormat(undefined, {
weekday: 'short', day: 'numeric', month: 'short', timeZone: 'UTC',
});
const shiftDay = (ymd) => shiftFmt.format(new Date(`${ymd}T00:00:00Z`));
function shiftCell(m) {
if (m.on_call) {
return h('span', { text: m.next_shift ? `today, then ${shiftDay(m.next_shift)}` : 'today' });
}
return m.next_shift
? h('span', { text: shiftDay(m.next_shift) })
: h('span', { text: 'not scheduled' });
}
function membersCard() {
const members = data.members || [];
const owners = members.filter((m) => m.role === 'owner').length;
const rows = members.map((m) => {
const lastOwner = m.role === 'owner' && owners === 1;
return h('tr', {},
h('td', {}, memberBadge(m)),
h('td', { class: 'wrap' },
h('strong', { text: m.username }),
m.user_id === myID() && h('span', { class: 'muted small', text: ' (you)' }),
m.problem && h('div', { class: 'target-problem', text: m.problem })),
h('td', { class: 'muted small', text: m.role }),
h('td', { class: 'muted small' }, shiftCell(m)),
h('td', { class: 'muted small' }, timeCell(m.last_active_at)),
h('td', { class: 'muted small' },
h('span', { title: when(m.joined_at), text: ago(m.joined_at) })),
h('td', {}, isOwner() && h('div', { class: 'row-actions' },
h('button', {
class: 'btn-sm', type: 'button', text: 'Edit', onclick: () => openEditMember(m),
}),
h('button', {
class: 'btn-sm danger', type: 'button', text: 'Remove',
disabled: lastOwner,
title: lastOwner ? 'A team needs an owner. Make somebody else one first.' : null,
onclick: async () => {
if (!(await confirm({
title: `Remove ${m.username}?`,
text: 'They lose access to this team. Rota days already assigned to them are not '
+ 'changed, so reassign those from the Rota tab.',
confirmLabel: 'Remove',
danger: true,
}))) return;
act(() => api.removeTeamMember(teamID, m.user_id));
},
}))),
);
});
return h('div', { class: 'card' },
h('div', { class: 'card-head' },
h('h2', { text: 'Members' }),
isOwner() && h('button', {
class: 'btn', type: 'button', text: 'Add member', onclick: openAddMember,
})),
h('p', { class: 'muted small' },
'Owners set up the team; members work its incidents. Somebody who can’t be ',
'paged is worth fixing before their next shift.'),
members.length
? h('div', { class: 'table-scroll' },
h('table', { class: 'admin-table status-table' },
h('thead', {}, h('tr', {},
h('th', { text: 'Status' }), h('th', { text: 'Member' }), h('th', { text: 'Role' }),
h('th', { text: 'Rota' }), h('th', { text: 'Last active' }), h('th', { text: 'Joined' }),
h('th'))),
h('tbody', {}, rows)))
: h('p', { class: 'muted', text: 'Nobody is in this team.' }),
);
}
// One sheet for both jobs a member's row has: who, and as what. Adding is
// choosing a person and a role; editing is the same with the person fixed. The
// API is one call either way — POST upserts the role.
function openMemberSheet({ title, submit, person, role, run }) {
const roleSelect = h('select', {},
...['member', 'owner'].map((r) => h('option', { value: r, text: r, selected: r === role })));
const problem = h('p', { class: 'load-error', hidden: true });
const form = h('form', { class: 'stacked-form' },
person.label,
h('label', {}, 'Role ', roleSelect),
problem,
h('div', { class: 'sheet-actions' },
h('button', { class: 'btn', type: 'button', text: 'Cancel', onclick: () => closeSheet(false) }),
h('button', { class: 'btn btn-primary', type: 'submit', text: submit })));
form.addEventListener('submit', async (e) => {
e.preventDefault();
try {
await run(person.userID(), roleSelect.value);
} catch (err) {
problem.textContent = err.message;
problem.hidden = false;
return;
}
closeSheet(true);
refresh();
});
openSheet(() => [h('h2', { class: 'sheet-title', text: title }), form]);
}
function openAddMember() {
const inTeam = new Set((data.members || []).map((m) => m.user_id));
const candidates = (data.users || []).filter((u) => !inTeam.has(u.id) && !u.disabled_at);
const pick = h('select', {},
...candidates.map((u) => h('option', { value: String(u.id), text: u.username })));
const role = h('select', {},
h('option', { value: 'member', text: 'member' }),
h('option', { value: 'owner', text: 'owner' }));
const form = h('form', { class: 'inline-form' }, pick, role,
h('button', { class: 'btn', type: 'submit', text: 'Add' }));
form.addEventListener('submit', (e) => {
e.preventDefault();
act(() => api.addTeamMember(teamID, Number(pick.value), role.value));
openMemberSheet({
title: 'Add member', submit: 'Add member', role: 'member',
person: {
label: candidates.length
? h('label', {}, 'Person ', pick)
: h('p', { class: 'muted', text: 'Everybody with an account is already in this team.' }),
userID: () => Number(pick.value),
},
run: (userID, role) => {
if (!userID) throw new Error('Nobody to add');
return api.addTeamMember(teamID, userID, role);
},
});
}
return h('div', { class: 'card' },
h('h2', { text: 'Members' }),
h('table', { class: 'admin-table' }, h('tbody', {}, rows)),
isOwner() && candidates.length > 0 && form,
);
function openEditMember(m) {
openMemberSheet({
title: `Edit ${m.username}`, submit: 'Save', role: m.role,
person: { label: h('p', { class: 'muted small', text: m.username }), userID: () => m.user_id },
run: (userID, role) => api.addTeamMember(teamID, userID, role),
});
}
// --- plumbing --------------------------------------------------------------
@@ -735,14 +1169,12 @@ function membersCard() {
// act runs a write and reloads. Errors are shown rather than thrown away: a
// 409 from the last-owner guard or the schedule's conflict rule is the server
// explaining itself, and the reader needs to see it.
async function act(fn, { resetDraft = false } = {}) {
async function act(fn) {
try {
await fn();
error = null;
if (resetDraft) draft = null;
} catch (err) {
error = err.message;
}
if (!resetDraft) draft = null;
await refresh();
}
+14
View File
@@ -34,15 +34,21 @@ export function clear(el, ...children) {
const ICONS = {
back: ['M15 18l-6-6 6-6'],
more: ['M5 12h.01M12 12h.01M19 12h.01'],
queueList: ['M4 6h16M4 12h16M4 18h10'],
calendar: ['rect:3.5,5,17,15,2', 'M3.5 10h17M8 3v4M16 3v4'],
team: ['circle:9,8,3', 'circle:17,9,2.5', 'M3 19a6 6 0 0 1 12 0M15 19a5 5 0 0 1 6-4'],
shield: ['M12 3l7 3v6c0 4-3 7-7 9-4-2-7-5-7-9V6z'],
check: ['M5 12.5l4.5 4.5L19 7'],
checkCircle: ['M8 12.5l3 3 5-6', 'circle:12,12,9'],
undo: ['M9 14L4 9l5-5', 'M4 9h10a6 6 0 0 1 0 12h-3'],
user: ['circle:12,8,3.5', 'M5 20a7 7 0 0 1 14 0'],
clock: ['circle:12,12,9', 'M12 7v5l3 2'],
chart: ['M4 20h16M7 20v-7M12 20V6M17 20v-10'],
bell: ['M6 16V11a6 6 0 0 1 12 0v5l1.5 2h-15z', 'M10 20.5a2 2 0 0 0 4 0'],
note: ['M5 4h14v12l-4 4H5z', 'M15 20v-4h4', 'M9 9h6M9 13h4'],
archive: ['M3.5 5h17v4h-17z', 'M5 9v10h14V9', 'M10 13h4'],
flag: ['M5 21V4', 'M5 4h11l-2 4 2 4H5'],
copy: ['rect:9,9,11,11,2', 'M5 15V6a2 2 0 0 1 2-2h9'],
trash: ['M4 7h16', 'M9 7V4h6v3', 'M6 7l1 13h10l1-13'],
chevronLeft: ['M15 18l-6-6 6-6'],
chevronRight: ['M9 6l6 6-6 6'],
@@ -64,6 +70,14 @@ export function icon(name, cls = 'icon') {
node.setAttribute('cx', cx);
node.setAttribute('cy', cy);
node.setAttribute('r', r);
} else if (d.startsWith('rect:')) {
const [x, y, w, hgt, rx] = d.slice(5).split(',');
node = document.createElementNS(SVG, 'rect');
node.setAttribute('x', x);
node.setAttribute('y', y);
node.setAttribute('width', w);
node.setAttribute('height', hgt);
if (rx) node.setAttribute('rx', rx);
} else {
node = document.createElementNS(SVG, 'path');
node.setAttribute('d', d);
+26
View File
@@ -0,0 +1,26 @@
package web
import (
"io/fs"
"strings"
"testing"
)
func TestStatsPageIsEmbedded(t *testing.T) {
sub, err := fs.Sub(files, "static")
if err != nil {
t.Fatal(err)
}
index, err := fs.ReadFile(sub, "index.html")
if err != nil {
t.Fatal(err)
}
for _, want := range []string{`id="view-stats"`, `data-section="stats"`} {
if !strings.Contains(string(index), want) {
t.Errorf("index.html lacks %s", want)
}
}
if _, err := fs.Stat(sub, "js/stats.js"); err != nil {
t.Errorf("js/stats.js not embedded: %v", err)
}
}