4 Commits

Author	SHA1	Message	Date
Niklas Ye	94dec19976	Räkna en tom incidentlista som noll i stället för som ett fel ... SUM över noll rader är NULL i SQLite, inte 0. handleStatsIncidents läste de tre statusräknarna rakt in i int64, så i samma stund som filtret inte matchade någon rad föll skanningen på "converting NULL to int64 is unsupported" och hela /api/stats/incidents svarade 500. COUNT(*) ger däremot 0 utan knot, vilket är precis varför felet inte syns förrän tabellen töms — det är det enda uttrycket i satsen som klarar noll rader. Filtret är alltid på: statsFilter lägger på archived_at IS NULL (923fc8b, flyttat hit i 279ef6c). En installation som varit tyst ett tag arkiverar därmed sig själv in i felet. Det är sluttillståndet för en lugn vecka, inte ett kantfall, och klustret står i det nu. Symptomet pekade åt fel håll. terdut-tui hämtar listan och statistiken i samma uppdatering, så incidentvyn såg trasig ut medan /api/incidents svarade 200 med []. Loggen i klustret visar de två anropen bredvid varandra, det ena grönt och det andra rött. Ingen ändring i terdut-tui behövs: dess ListIncidents är oförändrad sedan 0.7.2 och skickar samma parametrar som förut. handleStatsAlerts bar samma fel och rättas likadant, innan någon hittar det på samma sätt. COALESCE i SQL i stället för sql.NullInt64 i Go, eftersom jämförelserna redan bor i satserna här (severityRankSQL, 279ef6c). mtta_seconds och mttr_seconds lämnas medvetet utan COALESCE. null betyder "inget att mäta ännu" och 0 skulle läsas som "omedelbart" — två olika påståenden, och testet från 279ef6c låser fast skillnaden. Claude-Session: https://claude.ai/code/session_01S7R4gWTz5wh5xCY4nCSJjN	2026-09-01 21:47:39 +02:00
Niklas Ye	279ef6cf8b	Turn incoming alerts into incidents ... The alerts row was both Alertmanager's record and the human work queue, and the two have different owners. The webhook upsert rewrites that row on every notification; acknowledgement, comments and archiving were columns on it that the upsert happened not to touch. So an alert that resolved and re-fired days later still read as acknowledged by whoever acked the first occurrence — the ack outlived the thing it referred to. Nothing recorded transitions either: rows are mutated in place, so there was no timeline and no way to compute how long anything took. Alerts are now read-only signal records with two states, and incidents are the work item: triggered, acknowledged or resolved, with an assignee, a snooze, notes and an append-only timeline. Many alerts map to one incident, and a new occurrence opens a new incident, which is what makes a stale ack impossible rather than merely unlikely. Correlation uses Alertmanager's own groupKey. It already grouped the alerts according to the group_by routing tree the operator configured and sends the result on every webhook, where it was being discarded; adopting it means changing group_by in alertmanager.yml changes correlation here, with no second grouping scheme to configure and keep in sync. An incident opens only when an alert transitions into firing — an unseen fingerprint, a newer startsAt, or a resolved alert starting again. The unchanged notifications Alertmanager re-sends every repeat_interval are none of those. That rule is what lets manual resolution be terminal: without it, closing an incident by hand would be undone by the next re-send of an alert that never stopped firing, and the button would be a lie. Snooze covers the "not now" case instead. Incidents otherwise resolve by cascade, once every alert under them has stopped firing, whether by webhook or by expiry. New incidents are assigned to whoever holds today's schedule entry. The schedule table has existed since the first release with nothing reading it. Also here, following from the split: - Incident severity is a high-water mark over its alerts, never lowered. An incident that hit critical was a critical incident, and downgrading a live one would demote it in the queue while the work is still open. - /api/stats/incidents reports MTTA and MTTR, null rather than zero until there is something to average. Neither was computable before. - Alert archiving becomes sweeper-only housekeeping; the archive people interact with is the incident's. Breaking: the alert acknowledge, archive and comment endpoints are gone, and the alert object drops the acknowledgement fields and gains incident_id. The README maps each removed endpoint to its replacement. Migration 008 backfills an incident per existing alert, archived ones included so no comment is orphaned, carrying acknowledgements across and turning comments into timeline notes. Both documented alert contracts are untouched: received_at still advances on every accepted payload, re-sends included, and resolution_source still says how much to trust ends_at. The upsert is byte-for-byte what it was, now running inside the ingest transaction.	2026-07-30 17:02:13 +02:00
Niklas Ye	42e846f876	Expire stale firing alerts ... A resolved webhook was the only path out of the firing state, so a notification that was dropped, silenced, or lost to a restart pinned an alert as firing forever — Prometheus showed it resolved while terdut-server kept listing it. The archiver only ever touched resolved alerts, and both the list and stats queries compared status with plain equality, so a stale row was indistinguishable from a live one. A sweeper pass now resolves firing alerts on either of two signals: the ends_at watermark Alertmanager sets on outgoing firing notifications has passed (plus a grace period for clock skew), or no webhook has refreshed the alert within TERDUT_STALE_AFTER (default 6h, above Alertmanager's 4h repeat_interval). Such alerts get resolution_source = 'expiry', distinguishing them from a real 'alertmanager' resolve. Two related webhook bugs fixed alongside: - The upsert had no ordering guard, so a retried firing notification arriving after the resolved one resurrected the alert. Payloads for an older alert instance are now discarded: a stale retry carries the same startsAt, a genuine re-fire a newer one. - archived_at was never cleared on re-fire, leaving a re-fired alert archived and invisible in the default list. Stats now exclude archived alerts to match the default list view; this lowers historical firing/resolved totals. The chart exposes both sweeper durations via sweeper.staleAfter and sweeper.archiveAfter.	2026-07-28 11:49:39 +02:00
Niklas Ye	923fc8bf9c	Stage 6: alert statistics endpoints ... - GET /api/stats/alerts — total/firing/resolved counts - GET /api/stats/alerts/top — most frequent alert names (?limit, default 10) - GET /api/stats/alerts/by-hour — counts for all 24 hours (zeros filled in) - GET /api/stats/alerts/by-day — counts for all 7 days with names (zeros filled in) - All endpoints accept optional ?from/?to (YYYY-MM-DD) to filter by received_at	2026-05-20 22:31:18 +02:00