# Terminal Duty (terdut-server)

Incident management for teams that already run Prometheus Alertmanager. Point Alertmanager at it,
and alerts become incidents that get assigned to whoever is on call, paged, escalated when nobody
answers, and tracked to resolution. One binary, one Postgres.

![The incident queue with an incident open](./docs/images/queue-light.png)

## Highlights

- **Alertmanager-native.** Receives Alertmanager webhooks directly, with no adapter, and groups
  alerts into incidents by Alertmanager's own `groupKey`. An incident opens on a new occurrence,
  not on every re-send. [Details](./docs/incidents.md)
- **A real incident workflow.** Acknowledge, assign, snooze, add notes, resolve and archive, with a
  full timeline of who did what and when. Several clusters can feed one team without
  cross-talk. [Details](./docs/incidents.md)
- **On-call rota.** Each team keeps its own rota, and new incidents go to whoever is on call.
  [Web UI](./docs/web-ui.md)
- **Pages that escalate.** Push notifications through [ntfy](https://ntfy.sh), with an
  Acknowledge button right in the notification, and escalation ladders that move on to the next
  level when nobody answers. [Notifications](./docs/notifications.md) ·
  [Escalation](./docs/escalation.md)
- **Notices when the alerts stop.** Dead man's switches turn the absence of a heartbeat such as
  `Watchdog` into an incident. [Details](./docs/dead-mans-switch.md)
- **Teams.** Every team owns its queue, rota, escalation, alert sources and switches; people see
  only the teams they belong to.
- **Stats.** Incident counts, mean time to acknowledge and resolve, and alert frequency by name,
  hour and day. [Web UI](./docs/web-ui.md)
- **Single sign-on.** OpenID Connect with group-to-team and administrator mapping, and a device
  flow so a terminal client can sign in through the browser. [Details](./docs/single-sign-on.md)
- **Built for the phone first.** The web UI is served by the same binary, follows the system's
  dark mode, and can be added to the home screen.
- **An API for everything.** A REST API with API keys and service accounts for automation.
  [Reference](./docs/api.md)
- **Easy to run.** A scratch container image, a Helm chart, and
  [terdut-operator](https://git.ryuvia.com/niklas/terdut-operator) if you want teams and
  escalation as Kubernetes objects. [Deployment](./docs/deployment.md)

## Screenshots

| | |
|---|---|
| ![An incident with a note on its timeline](./docs/images/incident-note.png) | ![The queue in dark mode](./docs/images/queue-dark.png) |
| **Work an incident**: acknowledge, assign, snooze, note, resolve | **Dark mode**, following the system |
| ![The on-call page](./docs/images/oncall.png) | ![An escalation ladder](./docs/images/team-escalation.png) |
| **On call**: who holds the pager now, and the week ahead | **Escalation**: who is paged next, and when |
| ![Statistics](./docs/images/stats.png) | ![The alert feed](./docs/images/alerts.png) |
| **Stats**: MTTA, MTTR and what fires most | **Alerts**: the raw feed behind the incidents |

On a phone the queue and the incident page are the same interface, with a sticky action bar:

<p>
  <img src="./docs/images/mobile-queue.png" alt="The queue on a phone" width="240">
  <img src="./docs/images/mobile-incident.png" alt="An incident on a phone" width="240">
</p>

## Quick start

You need Go 1.25+ and a Postgres 14+ the server can reach.

```bash
git clone https://git.ryuvia.com/niklas/terdut-server
cd terdut-server
export TERDUT_DB_DSN='postgres://terdut:secret@localhost:5432/terdut?sslmode=disable'
go run ./cmd/terdut
```

The server creates its schema on startup and listens on `:8080`. Create the first user, an
administrator, while no user exists yet:

```bash
curl -X POST http://localhost:8080/api/bootstrap \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "email": "admin@example.com", "password": "<at least 10 characters>"}'
```

Sign in at <http://localhost:8080> with that username and password. The response also carries an
API key, shown **once**, for scripts:

```bash
curl -H "Authorization: Bearer $KEY" http://localhost:8080/api/users
```

Then create a team, add an alert source to get a webhook URL, and point Alertmanager's webhook
receiver at it: see [Alertmanager configuration](./docs/alertmanager.md).

## Documentation

The [documentation index](./docs/README.md) lists everything. The main pages:

| | |
|---|---|
| [Deployment](./docs/deployment.md) | Docker, Helm chart, database, backups, the operator |
| [Configuration](./docs/configuration.md) | Environment variables and settings |
| [Alertmanager configuration](./docs/alertmanager.md) | Routes, keys, webhooks |
| [Alerts and incidents](./docs/incidents.md) | Correlation, lifecycle, on-call assignment |
| [Single sign-on](./docs/single-sign-on.md) | OIDC and the terminal device flow |
| [API reference](./docs/api.md) | Every endpoint |
| [Development and releasing](./docs/development.md) | Tests, CI gate, release pipeline |

## Related

- [terdut-tui](https://git.ryuvia.com/niklas/terdut-tui): a terminal client for the same server.
- [terdut-operator](https://git.ryuvia.com/niklas/terdut-operator): a Kubernetes operator that
  runs the server and manages teams, escalation, switches and alert sources as objects.

## License

See [LICENSE](./LICENSE).