niklas/terdut-operator
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

TerdutTeam can wedge permanently in a 403 retry loop when its service-account mint succeeds but credential persistence fails #3

New Issue
Open
opened 2026-10-02 19:10:59 +00:00 by niklas · 0 comments
niklas commented 2026-10-02 19:10:59 +00:00
Owner
Copy Link
Copy Source

Summary

Caught live while verifying examples/demo/run-demo.sh against a genuinely
fresh kind cluster (not a re-run, not a contrived scenario): terdutteam-platform
got permanently stuck Ready: False while its sibling terdutteam-payments
(same manifest shape, same reconcile loop, same instant) converged fine.

Evidence, from the live cluster

service_accounts table — exactly one row for platform, proving the
original mint succeeded:

 id |                         name                         |  scope   | team_id | created_by | created_at
----+------------------------------------------------------+----------+---------+------------+------------
  1 | terdut-operator                                      | instance |         |          1 | 1790967849
  2 | terdut-team.terdut-operator-demo.terdutteam-payments | team     |       2 |            | 1790967863
  3 | terdut-team.terdut-operator-demo.terdutteam-platform | team     |       3 |            | 1790967863

service_account_keys — exactly one key per account, including platform's:

 id | service_account_id |  name   | created_at
----+--------------------+---------+------------
  1 |                  1 | initial | 1790967849
  2 |                  2 | initial | 1790967863
  3 |                  3 | initial | 1790967863

So mintTeamCredential's CreateTeamServiceAccount call for platform
returned 201 and minted key id 3 — same as payments. Yet the operator's
own log shows, at the very same timestamp:

2026-10-02T19:04:23Z ERROR Reconciler error {"...","error": "Secret \"terdut-operator-demo.terdutteam-platform-team-credentials\" not found"}
2026-10-02T19:04:23Z ERROR Reconciler error {"...","error": "POST /api/service-accounts/3/keys (adopting after 409): server returned 403: team owner, system administrator, or the account itself may rotate its key"}
```//
repeated indefinitely (dozens of times across the run, no backoff cutoff,
no terminal condition/event distinguishing this from a transient error).

## Root cause

1. `mintTeamCredential` (`internal/controller/terdutteam_bootstrap.go`)
   successfully creates the team-scoped service account + its first key
   (confirmed: row id 3 / key id 3 both exist, `created_at` matching
   payments' own successful run to the second).
2. Something between that success and `writeOperatorSecret` persisting the
   credentials Secret — or between that and `TerdutTeam.status.CredentialsSecretRef`
   actually being saved — is lost (the `Secret ... not found` error is the
   visible symptom). `terdutteam-payments` didn't hit this; it looks
   stochastic, not deterministic.
3. Because `status.CredentialsSecretRef` was never set, the *next* reconcile
   believes no credential exists yet and calls `CreateTeamServiceAccount`
   again → `409` (the account from step 1 already exists) → falls into the
   documented adopt-on-409 path → `POST /api/service-accounts/{id}/keys` to
   mint a fresh key on the now-existing account.
4. That call 403s: `callerMayManageServiceAccount`
   (`terdut-server/internal/api/service_accounts.go`) grants key-rotation
   only to a human admin, the target team's human owner, or the account
   itself. The caller here is the **instance-scoped** service account acting
   on a **different**, team-scoped account — none of the three conditions
   hold, confirmed against source:
   ```go
   func callerMayManageServiceAccount(ctx context.Context, sa models.ServiceAccount) bool {
       if callerIsAdmin(ctx) { return true }                       // false: no human caller
       if sa.TeamID != nil && callerOwnsTeam(ctx, *sa.TeamID) { return true } // false: instance account owns no team
       if self, ok := serviceAccountFromContext(ctx); ok && self.id == sa.ID { return true } // false: different account
       return false
   }
  1. Every reconcile from here on repeats steps 3–4 forever: 409 then 403,
    with no way to ever reach a 201 (create) or a successful adopt (mint).
    Permanent, unrecoverable Ready: False for that TerdutTeam and
    everything teamRef-ing it (its TerdutEscalationRule,
    TerdutDeadmanSwitch, TerdutAlertSource all sat on WaitingForTeam
    for the rest of the run).

Why this matters beyond the one unlucky reconcile

This is the same underlying gap as #23 on terdut-server (authorization
logic keyed on userFromContext/callerIsAdmin, which a service-account
caller can never satisfy) — but #23 only blocks admin-UI bootstrapping.
This one can permanently wedge a real production TerdutTeam the
instant its credential-persistence step has any hiccup at all (a pod
restart mid-reconcile, an apiserver conflict on the status update, anything
that separates "mint succeeded" from "Secret/status written" by even one
reconcile boundary) — with zero automatic recovery, since the fallback path
that's supposed to handle exactly this (DESIGN.md §5's adopt-on-409 rule)
is the thing that 403s.

Suggested fix directions (not prescriptive)

  • The real fix likely belongs in terdut-server (per #23): let an
    instance-scoped service account satisfy callerMayManageServiceAccount
    for any team-scoped account, the same way it's already trusted to
    create one.
  • Separately/additionally, the operator's own adopt-on-409 path has no
    terminal state distinct from "transient, keep requeuing" — a
    CredentialMintStateLost-style condition (mirroring TerdutServer's own
    bootstrapStateLostError handling for the exact same class of problem)
    would at least make this visible/diagnosable instead of a silent infinite
    error loop.

Repro

Not reliably deterministic on demand (didn't reproduce for terdutteam-payments
in the same run), but was hit on a genuinely first, fresh-cluster run of
examples/demo/run-demo.sh — no prior state, no re-run. Likely a timing
race around the Secret write / status update following a successful
service-account mint.

References

  • internal/controller/terdutteam_bootstrap.go — mintTeamCredential's
    create-then-adopt-on-409 flow.
  • terdut-server/internal/api/service_accounts.go —
    callerMayManageServiceAccount, handleCreateServiceAccountKey.
  • terdut-server/niklas#23 — the sibling authorization gap on the
    admin-settings/user-management routes.
  • DESIGN.md §5 (idempotent-create / adopt-on-conflict rule), §6 (bootstrap
    credential lifecycle).
## Summary Caught live while verifying `examples/demo/run-demo.sh` against a genuinely fresh kind cluster (not a re-run, not a contrived scenario): `terdutteam-platform` got permanently stuck `Ready: False` while its sibling `terdutteam-payments` (same manifest shape, same reconcile loop, same instant) converged fine. ## Evidence, from the live cluster `service_accounts` table — exactly one row for platform, proving the *original* mint succeeded: ``` id | name | scope | team_id | created_by | created_at ----+------------------------------------------------------+----------+---------+------------+------------ 1 | terdut-operator | instance | | 1 | 1790967849 2 | terdut-team.terdut-operator-demo.terdutteam-payments | team | 2 | | 1790967863 3 | terdut-team.terdut-operator-demo.terdutteam-platform | team | 3 | | 1790967863 ``` `service_account_keys` — exactly one key per account, including platform's: ``` id | service_account_id | name | created_at ----+--------------------+---------+------------ 1 | 1 | initial | 1790967849 2 | 2 | initial | 1790967863 3 | 3 | initial | 1790967863 ``` So `mintTeamCredential`'s `CreateTeamServiceAccount` call for platform returned `201` and minted key id 3 — same as payments. Yet the operator's own log shows, at the very same timestamp: ``` 2026-10-02T19:04:23Z ERROR Reconciler error {"...","error": "Secret \"terdut-operator-demo.terdutteam-platform-team-credentials\" not found"} 2026-10-02T19:04:23Z ERROR Reconciler error {"...","error": "POST /api/service-accounts/3/keys (adopting after 409): server returned 403: team owner, system administrator, or the account itself may rotate its key"} ```// repeated indefinitely (dozens of times across the run, no backoff cutoff, no terminal condition/event distinguishing this from a transient error). ## Root cause 1. `mintTeamCredential` (`internal/controller/terdutteam_bootstrap.go`) successfully creates the team-scoped service account + its first key (confirmed: row id 3 / key id 3 both exist, `created_at` matching payments' own successful run to the second). 2. Something between that success and `writeOperatorSecret` persisting the credentials Secret — or between that and `TerdutTeam.status.CredentialsSecretRef` actually being saved — is lost (the `Secret ... not found` error is the visible symptom). `terdutteam-payments` didn't hit this; it looks stochastic, not deterministic. 3. Because `status.CredentialsSecretRef` was never set, the *next* reconcile believes no credential exists yet and calls `CreateTeamServiceAccount` again → `409` (the account from step 1 already exists) → falls into the documented adopt-on-409 path → `POST /api/service-accounts/{id}/keys` to mint a fresh key on the now-existing account. 4. That call 403s: `callerMayManageServiceAccount` (`terdut-server/internal/api/service_accounts.go`) grants key-rotation only to a human admin, the target team's human owner, or the account itself. The caller here is the **instance-scoped** service account acting on a **different**, team-scoped account — none of the three conditions hold, confirmed against source: ```go func callerMayManageServiceAccount(ctx context.Context, sa models.ServiceAccount) bool { if callerIsAdmin(ctx) { return true } // false: no human caller if sa.TeamID != nil && callerOwnsTeam(ctx, *sa.TeamID) { return true } // false: instance account owns no team if self, ok := serviceAccountFromContext(ctx); ok && self.id == sa.ID { return true } // false: different account return false } ``` 5. Every reconcile from here on repeats steps 3–4 forever: `409` then `403`, with no way to ever reach a `201` (create) or a successful adopt (mint). **Permanent, unrecoverable `Ready: False`** for that `TerdutTeam` and everything `teamRef`-ing it (its `TerdutEscalationRule`, `TerdutDeadmanSwitch`, `TerdutAlertSource` all sat on `WaitingForTeam` for the rest of the run). ## Why this matters beyond the one unlucky reconcile This is the same underlying gap as #23 on `terdut-server` (authorization logic keyed on `userFromContext`/`callerIsAdmin`, which a service-account caller can never satisfy) — but #23 only blocks *admin-UI bootstrapping*. This one can permanently wedge a **real production `TerdutTeam`** the instant its credential-persistence step has any hiccup at all (a pod restart mid-reconcile, an apiserver conflict on the status update, anything that separates "mint succeeded" from "Secret/status written" by even one reconcile boundary) — with zero automatic recovery, since the fallback path that's supposed to handle exactly this (`DESIGN.md §5`'s adopt-on-409 rule) is the thing that 403s. ## Suggested fix directions (not prescriptive) - The real fix likely belongs in `terdut-server` (per #23): let an instance-scoped service account satisfy `callerMayManageServiceAccount` for any team-scoped account, the same way it's already trusted to *create* one. - Separately/additionally, the operator's own adopt-on-409 path has no terminal state distinct from "transient, keep requeuing" — a `CredentialMintStateLost`-style condition (mirroring `TerdutServer`'s own `bootstrapStateLostError` handling for the exact same class of problem) would at least make this visible/diagnosable instead of a silent infinite error loop. ## Repro Not reliably deterministic on demand (didn't reproduce for `terdutteam-payments` in the same run), but was hit on a genuinely first, fresh-cluster run of `examples/demo/run-demo.sh` — no prior state, no re-run. Likely a timing race around the Secret write / status update following a successful service-account mint. ## References - `internal/controller/terdutteam_bootstrap.go` — `mintTeamCredential`'s create-then-adopt-on-409 flow. - `terdut-server/internal/api/service_accounts.go` — `callerMayManageServiceAccount`, `handleCreateServiceAccountKey`. - `terdut-server/niklas#23` — the sibling authorization gap on the admin-settings/user-management routes. - `DESIGN.md` §5 (idempotent-create / adopt-on-conflict rule), §6 (bootstrap credential lifecycle).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
niklas (Niklas Ye)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: niklas/terdut-operator#3
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?