terdut-server/README.md

# Terminal Duty (terdut-server)

On-call alert management server for teams using Prometheus Alertmanager.

- Receives Alertmanager webhooks directly — no adapter needed
- Stores and queries alerts (acknowledge, comment)
- On-call schedule management (user-to-day assignments)
- Alert statistics (by status, by hour, by day)
- REST API with per-user API key authentication
- Single binary, SQLite storage — trivial to self-host

---

## Quick start

**Prerequisites:** Go 1.21+

```bash
git clone https://github.com/yeniklas/terdut-server
cd terdut-server
go run ./cmd/terdut
```

The server starts on `:8080` with a `terdut.db` file in the working directory.

### Create the first user

```bash
curl -X POST http://localhost:8080/api/bootstrap \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "email": "admin@example.com"}'
```

Save the `api_key.key` value from the response — it is shown **once only**.

Use it as a bearer token for all subsequent requests:

```bash
export KEY=<your-key>
curl -H "Authorization: Bearer $KEY" http://localhost:8080/api/users
```

### Docker

```bash
docker build -t terdut-server .
docker run -p 8080:8080 -v $(pwd)/data:/data \
  -e TERDUT_DB_PATH=/data/terdut.db \
  terdut-server
```

---

## Configuration

| Variable | Default | Description |
|---|---|---|
| `TERDUT_ADDR` | `:8080` | TCP address to listen on |
| `TERDUT_DB_PATH` | `terdut.db` | Path to the SQLite database file |
| `TERDUT_ARCHIVE_AFTER` | `168h` (7d) | How long a resolved alert stays in the default list before being auto-archived |
| `TERDUT_STALE_AFTER` | `6h` | How long a firing alert may go without a refreshing webhook before it is treated as resolved — **must exceed your Alertmanager `repeat_interval`** |

Durations use Go syntax (`30m`, `12h`, `168h`). An unparseable value falls back to the default.

In the Helm chart the two sweeper durations are set via `sweeper.staleAfter` and `sweeper.archiveAfter`.

---

## Alertmanager configuration

Add terdut-server as a webhook receiver in your `alertmanager.yml`:

```yaml
receivers:
  - name: terdut
    webhook_configs:
      - url: http://terdut-server:8080/api/alertmanager/webhook
        send_resolved: true

route:
  receiver: terdut
```

The webhook endpoint requires no authentication.

### Stale alert expiry

A resolved webhook is the only signal that an alert has stopped firing, so a
notification that is dropped, silenced, or lost to a restart would otherwise pin
that alert as firing forever. A background sweeper resolves firing alerts that
Alertmanager has stopped refreshing, using either signal:

- the `endsAt` watermark on the last notification has passed, or
- no webhook has refreshed the alert within `TERDUT_STALE_AFTER`.

Alertmanager re-sends firing notifications every `repeat_interval`, which is what
keeps a live alert fresh — so `TERDUT_STALE_AFTER` must be comfortably larger
than your `repeat_interval` (default 4h), or live alerts will be resolved
prematurely. Alerts resolved this way are marked `"resolution_source": "expiry"`
to distinguish them from a real Alertmanager resolve (`"alertmanager"`).

---

## API reference

### Authentication

All endpoints except `/api/bootstrap` and `/api/alertmanager/webhook` require:

```
Authorization: Bearer <api-key>
```

### Users

| Method | Path | Description |
|---|---|---|
| `POST` | `/api/bootstrap` | Create first user + API key (only works on empty DB) |
| `GET` | `/api/users` | List users |
| `POST` | `/api/users` | Create user `{"username","email"}` |
| `DELETE` | `/api/users/{id}` | Delete user (cascades to keys) |
| `POST` | `/api/users/{id}/api-keys` | Issue API key `{"name"}` — key shown once |
| `DELETE` | `/api/users/{id}/api-keys/{keyID}` | Revoke API key |

### Alert ingestion

| Method | Path | Description |
|---|---|---|
| `POST` | `/api/alertmanager/webhook` | Alertmanager v4 webhook receiver (no auth) |

### Alerts

| Method | Path | Description |
|---|---|---|
| `GET` | `/api/alerts` | List alerts. Filters: `?status=firing\|resolved`, `?name=`, `?archived=true`, `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD`, `?limit=` (default 50, max 500) |
| `GET` | `/api/alerts/{id}` | Get single alert |
| `POST` | `/api/alerts/{id}/acknowledge` | Acknowledge alert (stamps authed user + time) |
| `DELETE` | `/api/alerts/{id}/acknowledge` | Clear acknowledgement |
| `POST` | `/api/alerts/{id}/archive` | Archive alert (hides it from the default list) |
| `DELETE` | `/api/alerts/{id}/archive` | Un-archive alert |
| `GET` | `/api/alerts/{id}/comments` | List comments (chronological) |
| `POST` | `/api/alerts/{id}/comments` | Add comment `{"content"}` |
| `DELETE` | `/api/alerts/{id}/comments/{commentID}` | Delete own comment |

Archived alerts are hidden from `GET /api/alerts` unless `?archived=true` is
passed. Resolved alerts carry `resolution_source`: `"alertmanager"` for a real
resolved webhook, `"expiry"` when the sweeper inferred it (see
[Stale alert expiry](#stale-alert-expiry)).

### On-call schedule

| Method | Path | Description |
|---|---|---|
| `POST` | `/api/schedule` | Assign user to dates `{"user_id", "dates":["YYYY-MM-DD",...]}` — all-or-nothing |
| `GET` | `/api/schedule` | List entries. Filters: `?from=YYYY-MM-DD`, `?to=YYYY-MM-DD` |
| `GET` | `/api/schedule/current` | Today's on-call user (UTC), 404 if none |
| `DELETE` | `/api/schedule/{id}` | Remove schedule entry |

### Statistics

All stat endpoints accept optional `?from=YYYY-MM-DD` and `?to=YYYY-MM-DD` to filter by `received_at`. Archived alerts are excluded, matching the default alert list.

| Method | Path | Description |
|---|---|---|
| `GET` | `/api/stats/alerts` | `{total, firing, resolved}` counts |
| `GET` | `/api/stats/alerts/top` | Most frequent alert names. `?limit=` (default 10, max 100) |
| `GET` | `/api/stats/alerts/by-hour` | Count per hour-of-day (UTC), all 24 slots returned |
| `GET` | `/api/stats/alerts/by-day` | Count per day-of-week, all 7 slots with names returned |

---

## Development

```bash
go test ./...        # run all tests
go build ./...       # compile all packages
go run ./cmd/terdut  # run locally
```