Stage 3: TerdutEscalationRule + TerdutDeadmanSwitch

Both child CRDs resolve their own teamRef -> TerdutTeam.status via the new
shared resolveTeamAndClient helper (childref.go), never chaining up to
TerdutServer (DESIGN.md §5) -- TerdutTeam.status.serverEndpoint, added in
this same stage, is what makes that literally true.

TerdutEscalationRule: one PUT /api/teams/{id}/escalation per reconcile
(an upsert server-side, confirmed against source), resolving each "user"
target's username to a user_id via GET /api/users first and reporting
Ready: False, reason: UnknownUser if it doesn't resolve. No DELETE exists
for this resource, so its delete path PUTs an empty policy as the closest
available undo.

TerdutDeadmanSwitch: real create/update-in-place/delete, using
terdut-server v0.33.0's PUT (added specifically for this operator). No
unique-name constraint server-side, so idempotent-create here is
GET-list-and-match-by-name rather than adopt-on-409.

Extends tdclient with User/GetUserByUsername, the escalation request types
+ SetEscalation, and DeadmanSwitch + its CRUD methods. Also folds
ConditionTeamReady into the single shared ConditionReady constant, since
both were literally "Ready" and Stage 3 would otherwise have needed a
third same-valued constant.

internal/controller/terdutserver_controller_test.go's fakeTerdutServer
grows GET /api/users, PUT .../escalation, and the full dead man's switch
collection/item routes, replacing the old parseTeamPath/handleTeamByID
pair with a more general parseTeamSubPath/handleTeamSubPath dispatcher
that still covers every existing Stage 1/2 route unchanged.

make fmt lint test build all clean; envtest coverage for
internal/controller: 50.5% -> 71.7%.

config/crd/bases/terdut.ryuvia.com_terdutdeadmanswitches.yaml

@@ -0,0 +1,180 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.22.0
+  name: terdutdeadmanswitches.terdut.ryuvia.com
+spec:
+  group: terdut.ryuvia.com
+  names:
+    kind: TerdutDeadmanSwitch
+    listKind: TerdutDeadmanSwitchList
+    plural: terdutdeadmanswitches
+    singular: terdutdeadmanswitch
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.teamRef.name
+      name: Team
+      type: string
+    - jsonPath: .status.switchID
+      name: SwitchID
+      type: integer
+    - jsonPath: .status.conditions[?(@.type=="Ready")].status
+      name: Ready
+      type: string
+    - jsonPath: .status.conditions[?(@.type=="Ready")].reason
+      name: Reason
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: TerdutDeadmanSwitch is the Schema for the terdutdeadmanswitches
+          API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: spec defines the desired state of TerdutDeadmanSwitch
+            properties:
+              matcher:
+                description: |-
+                  matcher names the alerts this switch watches, e.g.
+                  "alertname=Watchdog,cluster=prod". One matcher per switch -- add
+                  another TerdutDeadmanSwitch instead of separating with ";"
+                  (terdut-server's own restriction, mirrored here so a bad spec is
+                  rejected at apply time).
+                minLength: 1
+                type: string
+                x-kubernetes-validations:
+                - message: 'one matcher per switch: add another TerdutDeadmanSwitch
+                    instead of separating with ;'
+                  rule: '!self.contains('';'')'
+              name:
+                description: |-
+                  name is optional, same as the API: left empty, terdut-server derives
+                  it from matcher's own canonical form, and that's what the
+                  idempotent-create lookup matches against too.
+                type: string
+              severity:
+                default: critical
+                enum:
+                - critical
+                - error
+                - warning
+                - info
+                type: string
+              teamRef:
+                description: |-
+                  TerdutTeamRef names the TerdutTeam this resource belongs to. Always
+                  same-namespace as the CR itself (DESIGN.md §1: only TerdutTeam.spec.serverRef
+                  crosses namespaces in v1) -- no namespace field, unlike TerdutServerRef.
+                properties:
+                  name:
+                    minLength: 1
+                    type: string
+                required:
+                - name
+                type: object
+              timeout:
+                description: timeout is a Go duration string, e.g. "15m".
+                minLength: 1
+                type: string
+            required:
+            - matcher
+            - teamRef
+            - timeout
+            type: object
+          status:
+            description: status defines the observed state of TerdutDeadmanSwitch
+            properties:
+              conditions:
+                items:
+                  description: Condition contains details for one aspect of the current
+                    state of this API Resource.
+                  properties:
+                    lastTransitionTime:
+                      description: |-
+                        lastTransitionTime is the last time the condition transitioned from one status to another.
+                        This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: |-
+                        message is a human readable message indicating details about the transition.
+                        This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    observedGeneration:
+                      description: |-
+                        observedGeneration represents the .metadata.generation that the condition was set based upon.
+                        For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+                        with respect to the current state of the instance.
+                      format: int64
+                      minimum: 0
+                      type: integer
+                    reason:
+                      description: |-
+                        reason contains a programmatic identifier indicating the reason for the condition's last transition.
+                        Producers of specific condition types may define expected values and meanings for this field,
+                        and whether the values are considered a guaranteed API.
+                        The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: status of the condition, one of True, False, Unknown.
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+                x-kubernetes-list-map-keys:
+                - type
+                x-kubernetes-list-type: map
+              observedGeneration:
+                format: int64
+                type: integer
+              switchID:
+                description: switchID is the server-side id.
+                format: int64
+                type: integer
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}

config/crd/bases/terdut.ryuvia.com_terdutescalationrules.yaml

@@ -0,0 +1,191 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.22.0
+  name: terdutescalationrules.terdut.ryuvia.com
+spec:
+  group: terdut.ryuvia.com
+  names:
+    kind: TerdutEscalationRule
+    listKind: TerdutEscalationRuleList
+    plural: terdutescalationrules
+    singular: terdutescalationrule
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.teamRef.name
+      name: Team
+      type: string
+    - jsonPath: .status.conditions[?(@.type=="Ready")].status
+      name: Ready
+      type: string
+    - jsonPath: .status.conditions[?(@.type=="Ready")].reason
+      name: Reason
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: TerdutEscalationRule is the Schema for the terdutescalationrules
+          API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: spec defines the desired state of TerdutEscalationRule
+            properties:
+              fallbackTopic:
+                type: string
+              levels:
+                items:
+                  description: |-
+                    EscalationLevel is one rung of the ladder: how long to wait, and who to
+                    page if nobody's acknowledged by then.
+                  properties:
+                    targets:
+                      items:
+                        description: |-
+                          EscalationTarget is one page within a level. username is required iff
+                          kind is "user" (terdut-server's own validation, internal/api/escalation.go's
+                          handleSetEscalation -- mirrored here as a CEL rule so a bad spec is
+                          rejected at apply time, not discovered on the next failed PUT).
+                        properties:
+                          kind:
+                            description: EscalationTargetKind is who one rung of the
+                              ladder pages.
+                            enum:
+                            - oncall
+                            - user
+                            type: string
+                          username:
+                            type: string
+                        required:
+                        - kind
+                        type: object
+                        x-kubernetes-validations:
+                        - message: username is required when kind is user
+                          rule: self.kind != 'user' || has(self.username)
+                        - message: username must not be set when kind is oncall
+                          rule: self.kind != 'oncall' || !has(self.username)
+                      minItems: 1
+                      type: array
+                    timeout:
+                      description: timeout is a Go duration string, e.g. "5m".
+                      minLength: 1
+                      type: string
+                  required:
+                  - targets
+                  - timeout
+                  type: object
+                minItems: 1
+                type: array
+              repeatCount:
+                format: int64
+                maximum: 10
+                minimum: 0
+                type: integer
+              teamRef:
+                description: |-
+                  TerdutTeamRef names the TerdutTeam this resource belongs to. Always
+                  same-namespace as the CR itself (DESIGN.md §1: only TerdutTeam.spec.serverRef
+                  crosses namespaces in v1) -- no namespace field, unlike TerdutServerRef.
+                properties:
+                  name:
+                    minLength: 1
+                    type: string
+                required:
+                - name
+                type: object
+            required:
+            - levels
+            - teamRef
+            type: object
+          status:
+            description: status defines the observed state of TerdutEscalationRule
+            properties:
+              conditions:
+                items:
+                  description: Condition contains details for one aspect of the current
+                    state of this API Resource.
+                  properties:
+                    lastTransitionTime:
+                      description: |-
+                        lastTransitionTime is the last time the condition transitioned from one status to another.
+                        This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: |-
+                        message is a human readable message indicating details about the transition.
+                        This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    observedGeneration:
+                      description: |-
+                        observedGeneration represents the .metadata.generation that the condition was set based upon.
+                        For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+                        with respect to the current state of the instance.
+                      format: int64
+                      minimum: 0
+                      type: integer
+                    reason:
+                      description: |-
+                        reason contains a programmatic identifier indicating the reason for the condition's last transition.
+                        Producers of specific condition types may define expected values and meanings for this field,
+                        and whether the values are considered a guaranteed API.
+                        The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: status of the condition, one of True, False, Unknown.
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+                x-kubernetes-list-map-keys:
+                - type
+                x-kubernetes-list-type: map
+              observedGeneration:
+                format: int64
+                type: integer
+            type: object
+        required:
+        - spec
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}

config/crd/bases/terdut.ryuvia.com_terdutteams.yaml

@@ -171,6 +171,13 @@ spec:
               observedGeneration:
                 format: int64
                 type: integer
+              serverEndpoint:
+                description: |-
+                  serverEndpoint is the resolved TerdutServer's base URL, resolved once
+                  here so no child controller (TerdutEscalationRule, TerdutDeadmanSwitch,
+                  TerdutAlertSource) ever needs its own RBAC on terdutservers just to
+                  find out where to send a request (DESIGN.md §5).
+                type: string
               teamID:
                 description: |-
                   teamID is the server-side id -- needed by every child object's

config/crd/kustomization.yaml

@@ -4,6 +4,8 @@
 resources:
 - bases/terdut.ryuvia.com_terdutservers.yaml
 - bases/terdut.ryuvia.com_terdutteams.yaml
+- bases/terdut.ryuvia.com_terdutescalationrules.yaml
+- bases/terdut.ryuvia.com_terdutdeadmanswitches.yaml
 # +kubebuilder:scaffold:crdkustomizeresource
 
 patches:

config/rbac/kustomization.yaml

@@ -22,6 +22,12 @@ resources:
 # default, aiding admins in cluster management. Those roles are
 # not used by the terdut-operator itself. You can comment the following lines
 # if you do not want those helpers be installed with your Project.
+- terdutdeadmanswitch_admin_role.yaml
+- terdutdeadmanswitch_editor_role.yaml
+- terdutdeadmanswitch_viewer_role.yaml
+- terdutescalationrule_admin_role.yaml
+- terdutescalationrule_editor_role.yaml
+- terdutescalationrule_viewer_role.yaml
 - terdutteam_admin_role.yaml
 - terdutteam_editor_role.yaml
 - terdutteam_viewer_role.yaml

config/rbac/role.yaml

@@ -55,6 +55,8 @@ rules:
 - apiGroups:
   - terdut.ryuvia.com
   resources:
+  - terdutdeadmanswitches
+  - terdutescalationrules
   - terdutservers
   - terdutteams
   verbs:
@@ -68,6 +70,8 @@ rules:
 - apiGroups:
   - terdut.ryuvia.com
   resources:
+  - terdutdeadmanswitches/finalizers
+  - terdutescalationrules/finalizers
   - terdutservers/finalizers
   - terdutteams/finalizers
   verbs:
@@ -75,6 +79,8 @@ rules:
 - apiGroups:
   - terdut.ryuvia.com
   resources:
+  - terdutdeadmanswitches/status
+  - terdutescalationrules/status
   - terdutservers/status
   - terdutteams/status
   verbs:

config/rbac/terdutdeadmanswitch_admin_role.yaml

@@ -0,0 +1,27 @@
+# This rule is not used by the project terdut-operator itself.
+# It is provided to allow the cluster admin to help manage permissions for users.
+#
+# Grants full permissions ('*') over terdut.ryuvia.com.
+# This role is intended for users authorized to modify roles and bindings within the cluster,
+# enabling them to delegate specific permissions to other users or groups as needed.
+
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  labels:
+    app.kubernetes.io/name: terdut-operator
+    app.kubernetes.io/managed-by: kustomize
+  name: terdutdeadmanswitch-admin-role
+rules:
+- apiGroups:
+  - terdut.ryuvia.com
+  resources:
+  - terdutdeadmanswitches
+  verbs:
+  - '*'
+- apiGroups:
+  - terdut.ryuvia.com
+  resources:
+  - terdutdeadmanswitches/status
+  verbs:
+  - get

config/rbac/terdutdeadmanswitch_editor_role.yaml

@@ -0,0 +1,33 @@
+# This rule is not used by the project terdut-operator itself.
+# It is provided to allow the cluster admin to help manage permissions for users.
+#
+# Grants permissions to create, update, and delete resources within the terdut.ryuvia.com.
+# This role is intended for users who need to manage these resources
+# but should not control RBAC or manage permissions for others.
+
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  labels:
+    app.kubernetes.io/name: terdut-operator
+    app.kubernetes.io/managed-by: kustomize
+  name: terdutdeadmanswitch-editor-role
+rules:
+- apiGroups:
+  - terdut.ryuvia.com
+  resources:
+  - terdutdeadmanswitches
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - terdut.ryuvia.com
+  resources:
+  - terdutdeadmanswitches/status
+  verbs:
+  - get

config/rbac/terdutdeadmanswitch_viewer_role.yaml

@@ -0,0 +1,29 @@
+# This rule is not used by the project terdut-operator itself.
+# It is provided to allow the cluster admin to help manage permissions for users.
+#
+# Grants read-only access to terdut.ryuvia.com resources.
+# This role is intended for users who need visibility into these resources
+# without permissions to modify them. It is ideal for monitoring purposes and limited-access viewing.
+
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  labels:
+    app.kubernetes.io/name: terdut-operator
+    app.kubernetes.io/managed-by: kustomize
+  name: terdutdeadmanswitch-viewer-role
+rules:
+- apiGroups:
+  - terdut.ryuvia.com
+  resources:
+  - terdutdeadmanswitches
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - terdut.ryuvia.com
+  resources:
+  - terdutdeadmanswitches/status
+  verbs:
+  - get

config/rbac/terdutescalationrule_admin_role.yaml

@@ -0,0 +1,27 @@
+# This rule is not used by the project terdut-operator itself.
+# It is provided to allow the cluster admin to help manage permissions for users.
+#
+# Grants full permissions ('*') over terdut.ryuvia.com.
+# This role is intended for users authorized to modify roles and bindings within the cluster,
+# enabling them to delegate specific permissions to other users or groups as needed.
+
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  labels:
+    app.kubernetes.io/name: terdut-operator
+    app.kubernetes.io/managed-by: kustomize
+  name: terdutescalationrule-admin-role
+rules:
+- apiGroups:
+  - terdut.ryuvia.com
+  resources:
+  - terdutescalationrules
+  verbs:
+  - '*'
+- apiGroups:
+  - terdut.ryuvia.com
+  resources:
+  - terdutescalationrules/status
+  verbs:
+  - get

config/rbac/terdutescalationrule_editor_role.yaml

@@ -0,0 +1,33 @@
+# This rule is not used by the project terdut-operator itself.
+# It is provided to allow the cluster admin to help manage permissions for users.
+#
+# Grants permissions to create, update, and delete resources within the terdut.ryuvia.com.
+# This role is intended for users who need to manage these resources
+# but should not control RBAC or manage permissions for others.
+
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  labels:
+    app.kubernetes.io/name: terdut-operator
+    app.kubernetes.io/managed-by: kustomize
+  name: terdutescalationrule-editor-role
+rules:
+- apiGroups:
+  - terdut.ryuvia.com
+  resources:
+  - terdutescalationrules
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - terdut.ryuvia.com
+  resources:
+  - terdutescalationrules/status
+  verbs:
+  - get

config/rbac/terdutescalationrule_viewer_role.yaml

@@ -0,0 +1,29 @@
+# This rule is not used by the project terdut-operator itself.
+# It is provided to allow the cluster admin to help manage permissions for users.
+#
+# Grants read-only access to terdut.ryuvia.com resources.
+# This role is intended for users who need visibility into these resources
+# without permissions to modify them. It is ideal for monitoring purposes and limited-access viewing.
+
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  labels:
+    app.kubernetes.io/name: terdut-operator
+    app.kubernetes.io/managed-by: kustomize
+  name: terdutescalationrule-viewer-role
+rules:
+- apiGroups:
+  - terdut.ryuvia.com
+  resources:
+  - terdutescalationrules
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - terdut.ryuvia.com
+  resources:
+  - terdutescalationrules/status
+  verbs:
+  - get

config/samples/kustomization.yaml

@@ -2,4 +2,6 @@
 resources:
 - terdut_v1alpha1_terdutserver.yaml
 - terdut_v1alpha1_terdutteam.yaml
+- terdut_v1alpha1_terdutescalationrule.yaml
+- terdut_v1alpha1_terdutdeadmanswitch.yaml
 # +kubebuilder:scaffold:manifestskustomizesamples

config/samples/terdut_v1alpha1_terdutdeadmanswitch.yaml

@@ -0,0 +1,15 @@
+apiVersion: terdut.ryuvia.com/v1alpha1
+kind: TerdutDeadmanSwitch
+metadata:
+  labels:
+    app.kubernetes.io/name: terdut-operator
+    app.kubernetes.io/managed-by: kustomize
+  name: terdutdeadmanswitch-sample
+spec:
+  teamRef:
+    name: terdutteam-sample
+  # name is optional -- left empty, terdut-server derives it from matcher's
+  # own canonical form (DESIGN.md §4.4).
+  matcher: "alertname=Watchdog"
+  timeout: 15m
+  severity: critical

config/samples/terdut_v1alpha1_terdutescalationrule.yaml

@@ -0,0 +1,25 @@
+apiVersion: terdut.ryuvia.com/v1alpha1
+kind: TerdutEscalationRule
+metadata:
+  labels:
+    app.kubernetes.io/name: terdut-operator
+    app.kubernetes.io/managed-by: kustomize
+  name: terdutescalationrule-sample
+spec:
+  # One per team (DESIGN.md §4.3) -- a second TerdutEscalationRule naming
+  # the same teamRef would simply clobber this one every reconcile, since
+  # there's no admission-time check for it in v1.
+  teamRef:
+    name: terdutteam-sample
+  repeatCount: 2
+  fallbackTopic: platform-fallback
+  levels:
+    # username is required iff kind is "user", and rejected otherwise --
+    # enforced at apply time via CEL (api/v1alpha1/terdutescalationrule_types.go).
+    - timeout: 5m
+      targets:
+        - kind: user
+          username: alice
+    - timeout: 10m
+      targets:
+        - kind: oncall