Stage 1: TerdutServer full lifecycle (Deployment, Service, both database
CI / test (push) Successful in 1m46s

paths, self-registration bootstrap)

Replaces the bring-your-own-only Stage 1 (commit 1be7cf2) wholesale, per
the redesign in the previous two commits: the operator creates every
server it manages, so self-registration (DESIGN.md §6) is the only
bootstrap path, and Deployment/Service/database management builds
together with it (ROADMAP.md Stage 1) rather than behind a separate
later stage.

Grounded in terdut-server's actual chart (charts/terdut-server/templates/
deployment.yaml, values.yaml), not reconstructed from DESIGN.md's
illustrative YAML alone -- env var names, the password-via-PGPASSWORD
convention, the Recreate deployment strategy, /healthz probes, and the
TERDUT_OPERATOR_MODE=true decision (always on here, unlike the chart's
default-off: every write this operator's own future controllers make
goes through a service account already) all match that source exactly.

- api/v1alpha1: full TerdutServerSpec (image, replicas, networking,
  database, sweeper, deadman, notify, oidc, passwordLogin, allowedTeams).
  spec.database is a oneOf (dsn xor postgresClusterRef) via CEL
  XValidation. No spec.credentialsSecretRef -- removed entirely in the
  prior redesign commit, not carried forward.
- internal/controller:
  - terdutserver_deployment.go: Deployment + Service via CreateOrUpdate,
    owned (OwnerReference), env built field-for-field against the chart.
  - terdutserver_database.go: both §8 paths. The Zalando path resolves
    the postgresql.acid.zalan.do CR by convention (database/role both
    "terdut", matching every DESIGN.md example) and only ever confirms
    its generated credentials Secret exists -- never reads the value,
    same "wire a secretKeyRef, don't read it" posture the DSN path takes.
    classifyClusterGetError is its own function specifically so the
    CRD-not-installed case (meta.IsNoMatchError) is unit-testable without
    a real client.
  - terdutserver_bootstrap.go: self-registration, checkpointed against
    both real crash windows (DESIGN.md §6 point 1) -- an admin-key
    checkpoint Secret, and adopt-via-GET+mint-new-key on a 409 from
    creating the service account. BootstrapStateLost is its own error
    type so Reconcile can route it to a condition instead of an infinite
    retry.
  - terdutserver_controller.go: ties it together -- finalizer add, DB
    resolution, Deployment/Service reconcile, wait for a ready replica,
    bootstrap, Ready/Bootstrapped/DatabaseReady conditions. Finalizer on
    delete only removes the generated Secrets: terdut-server's API can't
    delete a user or service account, only revoke keys, so there's
    nothing server-side to undo.
- internal/tdclient: added Bootstrap, CreateInstanceServiceAccount,
  GetServiceAccountByName, CreateServiceAccountKey, matching
  terdut-server's real handlers' request/response shapes (internal/api/
  users.go, service_accounts.go in that repo) field-for-field.
- Tests: envtest suite covering the full DSN-path lifecycle end to end
  (finalizer -> Deployment/Service -> simulated readiness -> real
  bootstrap against an httptest.Server fake), the adopt-on-409 recovery
  path, BootstrapStateLost, both Zalando outcomes (cluster not found;
  cluster + Secret found -> real DSN -> Ready), and deletion. A minimal
  test-only stub of the Zalando CRD (internal/controller/testdata) lets
  envtest create fixture objects without a real postgres-operator
  installed. 74.0%/44.7% coverage, 0 lint issues.
- Two things scoped down from §8's full ambition, called out in code and
  ROADMAP.md rather than silently dropped: no live watch on the
  Zalando-generated Secret for rotation (periodic resync notices
  eventually, not immediately), no Gateway API HTTPRoute creation from
  spec.networking (would add a new dependency; nothing about proving
  bootstrap works depends on external ingress existing). Both are
  near-term follow-ups.

Verified locally: make fmt lint test build all clean.
This commit is contained in:
Niklas Ye
2026-10-01 09:11:56 +02:00
parent fc68ee7256
commit 8064876cb1
16 changed files with 1872 additions and 306 deletions
+5 -1
View File
@@ -50,7 +50,11 @@ var _ = BeforeSuite(func() {
By("bootstrapping test environment")
testEnv = &envtest.Environment{
CRDDirectoryPaths: []string{filepath.Join("..", "..", "config", "crd", "bases")},
// testdata carries a minimal, test-only stub of Zalando
// postgres-operator's CRD (see its own file comment) so
// terdutserver_database_test.go can exercise the postgresClusterRef
// path without a real postgres-operator installed.
CRDDirectoryPaths: []string{filepath.Join("..", "..", "config", "crd", "bases"), "testdata"},
ErrorIfCRDPathMissing: true,
}
@@ -0,0 +1,153 @@
package controller
import (
"context"
"errors"
"fmt"
"net/http"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
terdutv1alpha1 "git.ryuvia.com/niklas/terdut-operator/api/v1alpha1"
"git.ryuvia.com/niklas/terdut-operator/internal/tdclient"
)
// credentialsSecretDataKey is the fixed data key every generated
// credentials Secret this controller writes uses — "whatever a human chose"
// only applied to the bring-your-own input this design removed (DESIGN.md
// §6); every Secret this controller itself generates uses this one key.
const credentialsSecretDataKey = "token"
// bootstrapStateLostError is DESIGN.md §6's one genuinely pathological
// case: a checkpointed admin credential was used and then lost before the
// lasting credential it was for could be persisted. Distinct from a plain
// error so Reconcile can route it to a Ready: False condition (the
// documented recovery is delete-and-recreate, not an automatic retry) rather
// than treating it as a transient reconcile failure.
type bootstrapStateLostError struct{ detail string }
func (e *bootstrapStateLostError) Error() string {
return fmt.Sprintf(
"server reports already bootstrapped, but neither status.credentialsSecretRef nor a "+
"checkpointed admin credential exist here: %s. This TerdutServer cannot recover a "+
"credential on its own; delete and recreate it", e.detail)
}
// reconcileBootstrap implements DESIGN.md §6 point 1's self-registration
// flow, checkpointed against the two real crash windows in it rather than
// leaving them as theoretical gaps. Only called once
// srv.Status.CredentialsSecretRef is nil and the Deployment has a ready
// replica.
func (r *TerdutServerReconciler) reconcileBootstrap(ctx context.Context, srv *terdutv1alpha1.TerdutServer) error {
adminKey, err := r.getOrCreateCheckpointedAdminKey(ctx, srv)
if err != nil {
return err
}
bc := r.NewClient(serviceURL(srv)).WithToken(adminKey)
instanceKey, err := r.getOrMintInstanceServiceAccountKey(ctx, bc)
if err != nil {
return err
}
credsName := credentialsSecretName(srv)
if err := r.writeSecret(ctx, credsName, instanceKey); err != nil {
return err
}
srv.Status.CredentialsSecretRef = &terdutv1alpha1.SecretKeyRef{Name: credsName, Key: credentialsSecretDataKey}
// Best-effort: the checkpoint has done its job. Leaving it behind on a
// delete failure here isn't a correctness problem (the next reconcile
// finds status.CredentialsSecretRef already set and never looks at the
// checkpoint again) — it would just be an unused Secret sitting around,
// cleaned up for real by the finalizer on delete.
checkpoint := &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: checkpointSecretName(srv), Namespace: r.OperatorNamespace}}
_ = r.Delete(ctx, checkpoint)
return nil
}
// getOrCreateCheckpointedAdminKey returns a usable admin key: from the
// checkpoint Secret if an earlier, interrupted attempt already got one, or
// freshly from /api/bootstrap, immediately checkpointed before it's used
// for anything else.
func (r *TerdutServerReconciler) getOrCreateCheckpointedAdminKey(ctx context.Context, srv *terdutv1alpha1.TerdutServer) (string, error) {
checkpointName := checkpointSecretName(srv)
var checkpoint corev1.Secret
err := r.Get(ctx, client.ObjectKey{Namespace: r.OperatorNamespace, Name: checkpointName}, &checkpoint)
switch {
case err == nil:
return string(checkpoint.Data[credentialsSecretDataKey]), nil
case !apierrors.IsNotFound(err):
return "", err
}
bc := r.NewClient(serviceURL(srv))
result, err := bc.Bootstrap(ctx, bootstrapUsername, bootstrapEmail)
if err != nil {
if statusErr, ok := errors.AsType[*tdclient.StatusError](err); ok && statusErr.Code == http.StatusForbidden {
// §1: this operator is the only thing that ever bootstraps a
// server it created, so a 403 here (no checkpoint, no
// status.credentialsSecretRef) means a prior reconcile already
// won this exact race and its checkpoint was lost afterward --
// the one case §6 doesn't try to paper over.
return "", &bootstrapStateLostError{detail: "/api/bootstrap returned 403"}
}
return "", fmt.Errorf("POST /api/bootstrap: %w", err)
}
if err := r.writeSecret(ctx, checkpointName, result.APIKey.Key); err != nil {
return "", fmt.Errorf("checkpointing admin key: %w", err)
}
return result.APIKey.Key, nil
}
// getOrMintInstanceServiceAccountKey mints the operator's own instance-
// scoped service account, or, if an earlier interrupted attempt already
// created it (409), adopts it and mints a fresh key rather than treating
// the conflict as an error (DESIGN.md §6 point 1, §5's general
// adopt-on-conflict rule).
func (r *TerdutServerReconciler) getOrMintInstanceServiceAccountKey(ctx context.Context, bc *tdclient.Client) (string, error) {
result, err := bc.CreateInstanceServiceAccount(ctx, serviceAccountName)
if err == nil {
return result.Key.Key, nil
}
statusErr, ok := errors.AsType[*tdclient.StatusError](err)
if !ok || statusErr.Code != http.StatusConflict {
return "", fmt.Errorf("POST /api/service-accounts: %w", err)
}
sa, err := bc.GetServiceAccountByName(ctx, serviceAccountName)
if err != nil {
return "", fmt.Errorf("GET /api/service-accounts?name=%s (adopting after 409): %w", serviceAccountName, err)
}
if sa == nil {
return "", fmt.Errorf("POST /api/service-accounts 409'd for %q but GET found nothing", serviceAccountName)
}
key, err := bc.CreateServiceAccountKey(ctx, sa.ID, "initial")
if err != nil {
return "", fmt.Errorf("POST /api/service-accounts/%d/keys (adopting after 409): %w", sa.ID, err)
}
return key.Key, nil
}
// writeSecret creates or replaces a Secret in the operator's own namespace
// holding one raw value under credentialsSecretDataKey.
func (r *TerdutServerReconciler) writeSecret(ctx context.Context, name, rawValue string) error {
secret := &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: r.OperatorNamespace},
Data: map[string][]byte{credentialsSecretDataKey: []byte(rawValue)},
}
if err := r.Create(ctx, secret); err != nil {
if apierrors.IsAlreadyExists(err) {
return r.Update(ctx, secret)
}
return err
}
return nil
}
+146 -58
View File
@@ -2,16 +2,20 @@ package controller
import (
"context"
"errors"
"fmt"
"time"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
logf "sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/controller-runtime/pkg/recorder"
@@ -21,41 +25,67 @@ import (
// resyncInterval is the periodic requeue on a successful reconcile (DESIGN.md
// §5's general rule) — it exists to catch drift from someone changing state
// directly against the server's API/UI, not from a missed watch event.
// directly against the server's API/UI, not from a missed watch event. It
// also doubles as the eventual-rotation-detection interval for the Zalando
// database path (§8 asks for a live Secret watch; this controller doesn't
// have one yet, so a rotated credential is noticed on the next resync
// rather than immediately — a known simplification, not a design decision).
const resyncInterval = 5 * time.Minute
// waitInterval is the requeue while waiting on an external condition to
// resolve (the credential Secret not existing yet, the server being
// unreachable) — shorter than resyncInterval, since these are expected to
// change sooner than "someone edited something out of band."
const waitInterval = 30 * time.Second
// resolve (the Deployment not ready yet, a Zalando cluster not found yet) —
// shorter than resyncInterval, since these are expected to change sooner
// than "someone edited something out of band."
const waitInterval = 15 * time.Second
// finalizerName cleans up the credentials Secret(s) this controller
// generates in the operator's own namespace on delete — the Deployment and
// Service are owned (OwnerReference, DESIGN.md §7) and need no finalizer of
// their own.
const finalizerName = "terdut.ryuvia.com/terdutserver"
// serviceAccountName is the name the operator registers itself under
// server-side (DESIGN.md §6) — a fixed, repo-wide constant, not a spec
// field: it names the automation, not anything about this one TerdutServer.
const serviceAccountName = "terdut-operator"
// bootstrapUsername/bootstrapEmail found the one human-shaped user every
// fresh install needs (terdut-server's handleBootstrap requires both).
// Nobody signs in as this user afterward — its only purpose is minting the
// admin key the controller immediately trades for a real service-account
// key — so these are fixed, not spec fields.
const (
bootstrapUsername = "terdut-operator-bootstrap"
bootstrapEmail = "bootstrap@terdut-operator.local"
)
// postgresqlGVK is the Zalando postgres-operator's CR (DESIGN.md §8).
// Resolved via unstructured rather than vendoring Zalando's own client, to
// keep this operator's dependency on it to "an optional CRD read" rather
// than a library — matches §9's "degrade gracefully if the CRD isn't
// installed" stance.
var postgresqlGVK = schema.GroupVersionKind{Group: "acid.zalan.do", Version: "v1", Kind: "postgresql"}
// TerdutServerReconciler reconciles a TerdutServer object.
//
// Stage 1 (ROADMAP.md): bootstrap/credentials only, bring-your-own path
// only. It never creates, updates, or deletes anything server-side or any
// Deployment/Service — it only reads a Secret the operator's own namespace
// already has and probes the server's /api/version. No finalizer: this
// controller owns nothing that needs cleaning up on delete (it never creates
// the credentials Secret itself, only ever adopts one a human already
// made) — revisit once self-registration (Stage 5) generates its own.
// Stage 1 (ROADMAP.md): full lifecycle. The operator creates and owns every
// TerdutServer it manages (DESIGN.md §1) — there is no adopt path.
type TerdutServerReconciler struct {
client.Client
Scheme *runtime.Scheme
// OperatorNamespace is where every credentials Secret this controller
// reads or writes lives (DESIGN.md §6) — never the TerdutServer's own
// generates lives (DESIGN.md §6) — never the TerdutServer's own
// namespace. Set from the POD_NAMESPACE downward-API env var in
// production (cmd/main.go); tests set it directly.
OperatorNamespace string
// Recorder emits the Kubernetes Events DESIGN.md §12 asks for on every
// externally-visible outcome. The events.k8s.io/v1 API, not the
// deprecated core/v1 one GetEventRecorderFor still returns.
// externally-visible outcome.
Recorder recorder.EventRecorder
// NewClient builds the terdut-server API client for a given endpoint.
// A field, not a direct tdclient.New call, so tests can substitute an
// NewClient builds the terdut-server API client for a given endpoint. A
// field, not a direct tdclient.New call, so tests can substitute an
// httptest.Server's client without a real network round trip. Defaults
// to tdclient.New via SetupWithManager.
NewClient func(endpoint string) *tdclient.Client
@@ -64,7 +94,10 @@ type TerdutServerReconciler struct {
// +kubebuilder:rbac:groups=terdut.ryuvia.com,resources=terdutservers,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=terdut.ryuvia.com,resources=terdutservers/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=terdut.ryuvia.com,resources=terdutservers/finalizers,verbs=update
// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch
// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=acid.zalan.do,resources=postgresqls,verbs=get;list;watch
func (r *TerdutServerReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
log := logf.FromContext(ctx)
@@ -77,77 +110,88 @@ func (r *TerdutServerReconciler) Reconcile(ctx context.Context, req ctrl.Request
return ctrl.Result{}, err
}
if srv.Spec.CredentialsSecretRef == nil {
return r.setNotReady(ctx, &srv,
terdutv1alpha1.ReasonCredentialsSecretRefRequired,
"spec.credentialsSecretRef is unset; self-registration bootstrap isn't "+
"implemented yet (Stage 5) — mint an instance-scoped service account "+
"with your own admin session (POST /api/service-accounts) and set "+
"this field to a Secret holding its key (DESIGN.md §6)")
if !srv.DeletionTimestamp.IsZero() {
return r.reconcileDelete(ctx, &srv)
}
ref := srv.Spec.CredentialsSecretRef
var secret corev1.Secret
secretKey := client.ObjectKey{Namespace: r.OperatorNamespace, Name: ref.Name}
if err := r.Get(ctx, secretKey, &secret); err != nil {
if apierrors.IsNotFound(err) {
return r.setNotReady(ctx, &srv,
terdutv1alpha1.ReasonCredentialsSecretNotFound,
fmt.Sprintf("Secret %q not found in namespace %q", ref.Name, r.OperatorNamespace))
if !controllerutil.ContainsFinalizer(&srv, finalizerName) {
controllerutil.AddFinalizer(&srv, finalizerName)
if err := r.Update(ctx, &srv); err != nil {
return ctrl.Result{}, err
}
// The Update above re-triggers a reconcile via the watch; nothing
// further to do on this pass.
return ctrl.Result{}, nil
}
dbEnv, dbErr := r.resolveDatabaseEnv(ctx, &srv)
if dbErr != nil {
return r.setNotReady(ctx, &srv, dbErr.reason, dbErr.message, waitInterval)
}
deploy, err := r.reconcileDeployment(ctx, &srv, dbEnv)
if err != nil {
return ctrl.Result{}, err
}
if err := r.reconcileService(ctx, &srv); err != nil {
return ctrl.Result{}, err
}
raw, ok := secret.Data[ref.Key]
if !ok || len(raw) == 0 {
meta.SetStatusCondition(&srv.Status.Conditions, metav1.Condition{
Type: terdutv1alpha1.ConditionDatabaseReady,
Status: metav1.ConditionTrue,
Reason: terdutv1alpha1.ReasonAdopted,
Message: "database resolved",
})
if deploy.Status.ReadyReplicas < 1 {
return r.setNotReady(ctx, &srv,
terdutv1alpha1.ReasonCredentialsSecretInvalid,
fmt.Sprintf("Secret %q has no data under key %q", ref.Name, ref.Key))
terdutv1alpha1.ReasonWaitingForDeployment,
fmt.Sprintf("Deployment %q has no ready replica yet", deploy.Name),
waitInterval)
}
newClient := r.NewClient
if newClient == nil {
newClient = tdclient.New
}
if _, err := newClient(srv.Spec.Endpoint).Version(ctx); err != nil {
return r.setNotReady(ctx, &srv,
terdutv1alpha1.ReasonServerUnreachable,
fmt.Sprintf("GET %s/api/version: %v", srv.Spec.Endpoint, err))
if srv.Status.CredentialsSecretRef == nil {
if err := r.reconcileBootstrap(ctx, &srv); err != nil {
if pending, ok := errors.AsType[*bootstrapStateLostError](err); ok {
return r.setNotReady(ctx, &srv, terdutv1alpha1.ReasonBootstrapStateLost, pending.Error(), waitInterval)
}
return ctrl.Result{}, err
}
}
srv.Status.CredentialsSecretRef = ref.DeepCopy()
meta.SetStatusCondition(&srv.Status.Conditions, metav1.Condition{
Type: terdutv1alpha1.ConditionBootstrapped,
Status: metav1.ConditionTrue,
Reason: terdutv1alpha1.ReasonAdopted,
Message: fmt.Sprintf("adopted spec.credentialsSecretRef (Secret %q, key %q)", ref.Name, ref.Key),
Message: fmt.Sprintf("credentials in Secret %q", srv.Status.CredentialsSecretRef.Name),
})
meta.SetStatusCondition(&srv.Status.Conditions, metav1.Condition{
Type: terdutv1alpha1.ConditionReady,
Status: metav1.ConditionTrue,
Reason: terdutv1alpha1.ReasonAdopted,
Message: "credentials adopted, server reachable",
Message: "deployment ready, database resolved, credentials bootstrapped",
})
srv.Status.ServiceName = srv.Name
srv.Status.ObservedGeneration = srv.Generation
if err := r.Status().Update(ctx, &srv); err != nil {
return ctrl.Result{}, err
}
if r.Recorder != nil {
r.Recorder.Eventf(&srv, nil, corev1.EventTypeNormal,
terdutv1alpha1.ReasonAdopted, terdutv1alpha1.ReasonAdopted,
"credentials adopted, server reachable")
r.Recorder.Eventf(&srv, nil, corev1.EventTypeNormal, terdutv1alpha1.ReasonAdopted, terdutv1alpha1.ReasonAdopted,
"terdut-server ready")
}
log.Info("TerdutServer ready", "endpoint", srv.Spec.Endpoint)
log.Info("TerdutServer ready", "name", srv.Name)
return ctrl.Result{RequeueAfter: resyncInterval}, nil
}
// setNotReady records Ready: False with reason/message on both conditions,
// fires a Warning event, and requeues after waitInterval — every "waiting on
// something external" exit from Reconcile goes through here so the
// condition/event/requeue shape can't drift between them.
// setNotReady records Ready: False with reason/message, fires a Warning
// event, and requeues after d. Every "waiting on something" exit from
// Reconcile goes through here so the condition/event/requeue shape can't
// drift between them.
func (r *TerdutServerReconciler) setNotReady(
ctx context.Context, srv *terdutv1alpha1.TerdutServer, reason, message string,
ctx context.Context, srv *terdutv1alpha1.TerdutServer, reason, message string, d time.Duration,
) (ctrl.Result, error) {
meta.SetStatusCondition(&srv.Status.Conditions, metav1.Condition{
Type: terdutv1alpha1.ConditionReady,
@@ -162,7 +206,31 @@ func (r *TerdutServerReconciler) setNotReady(
if r.Recorder != nil {
r.Recorder.Eventf(srv, nil, corev1.EventTypeWarning, reason, reason, message)
}
return ctrl.Result{RequeueAfter: waitInterval}, nil
return ctrl.Result{RequeueAfter: d}, nil
}
// reconcileDelete cleans up the credentials Secret(s) this controller
// generated in the operator's own namespace. The Deployment and Service are
// owned (OwnerReference, DESIGN.md §7) and need no attention here — normal
// GC handles them. There is no server-side "delete this install" call to
// make: bootstrap created a user and a service account, and terdut-server's
// API has no way to delete either (only to revoke individual keys), so
// there is nothing meaningful to undo there either.
func (r *TerdutServerReconciler) reconcileDelete(ctx context.Context, srv *terdutv1alpha1.TerdutServer) (ctrl.Result, error) {
if !controllerutil.ContainsFinalizer(srv, finalizerName) {
return ctrl.Result{}, nil
}
for _, name := range []string{checkpointSecretName(srv), credentialsSecretName(srv)} {
sec := &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: r.OperatorNamespace}}
if err := r.Delete(ctx, sec); err != nil && !apierrors.IsNotFound(err) {
return ctrl.Result{}, err
}
}
controllerutil.RemoveFinalizer(srv, finalizerName)
if err := r.Update(ctx, srv); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{}, nil
}
// SetupWithManager sets up the controller with the Manager.
@@ -175,6 +243,26 @@ func (r *TerdutServerReconciler) SetupWithManager(mgr ctrl.Manager) error {
}
return ctrl.NewControllerManagedBy(mgr).
For(&terdutv1alpha1.TerdutServer{}).
Owns(&appsv1.Deployment{}).
Owns(&corev1.Service{}).
Named("terdutserver").
Complete(r)
}
// --- naming ---
func checkpointSecretName(srv *terdutv1alpha1.TerdutServer) string {
return fmt.Sprintf("%s.%s-bootstrap-admin", srv.Namespace, srv.Name)
}
func credentialsSecretName(srv *terdutv1alpha1.TerdutServer) string {
return fmt.Sprintf("%s.%s-instance-credentials", srv.Namespace, srv.Name)
}
func serviceURL(srv *terdutv1alpha1.TerdutServer) string {
port := srv.Spec.Networking.ServicePort
if port == 0 {
port = 8080
}
return fmt.Sprintf("http://%s.%s.svc:%d", srv.Name, srv.Namespace, port)
}
@@ -2,43 +2,120 @@ package controller
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"sync"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
terdutv1alpha1 "git.ryuvia.com/niklas/terdut-operator/api/v1alpha1"
"git.ryuvia.com/niklas/terdut-operator/internal/tdclient"
)
const (
// unusedEndpoint is a syntactically valid URL no test here ever expects
// to actually be dialed (the cases using it fail before reaching the
// server-reachability check).
unusedEndpoint = "http://unused.invalid"
// tokenKey is the data key every test's credentials Secret uses.
tokenKey = "token"
)
// fakeTerdutServer reproduces the exact stateful semantics of
// /api/bootstrap, /api/service-accounts and /api/version that the
// bootstrap flow depends on (DESIGN.md §6, §11: "terdut-server's REST API
// is faked with a small httptest.Server per controller test ... matching
// the real handlers' request/response shapes"), including the 403-after-
// first-success and 409-on-name-conflict behavior the adopt-on-conflict
// recovery path exists for.
type fakeTerdutServer struct {
mu sync.Mutex
bootstrapped bool
bootstrap403 bool // force every /api/bootstrap call to 403, even the first
nextID int64
accounts map[string]int64 // name -> id
keyMints map[int64]int // id -> number of keys minted so far
}
// fakeTerdutServer is an httptest.Server standing in for terdut-server's
// GET /api/version, per DESIGN.md §11 ("terdut-server's REST API is faked
// with a small httptest.Server per controller test ... no real Postgres or
// real terdut-server binary needed for controller unit tests").
func fakeTerdutServer(version string) *httptest.Server {
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/api/version" {
w.WriteHeader(http.StatusNotFound)
func newFakeTerdutServer() (*fakeTerdutServer, *httptest.Server) {
f := &fakeTerdutServer{accounts: map[string]int64{}, keyMints: map[int64]int{}}
return f, httptest.NewServer(f)
}
func (f *fakeTerdutServer) ServeHTTP(w http.ResponseWriter, r *http.Request) {
f.mu.Lock()
defer f.mu.Unlock()
switch {
case r.URL.Path == "/api/version":
writeJSON(w, http.StatusOK, map[string]string{"version": "test"})
case r.URL.Path == "/api/bootstrap" && r.Method == http.MethodPost:
if f.bootstrapped || f.bootstrap403 {
writeJSON(w, http.StatusForbidden, map[string]string{"error": "bootstrap already completed"})
return
}
fmt.Fprintf(w, `{"version":%q}`, version) //nolint:errcheck
}))
f.bootstrapped = true
writeJSON(w, http.StatusCreated, tdclient.BootstrapResult{
APIKey: tdclient.APIKey{ID: 1, Name: "bootstrap", Key: "admin-key-raw"},
})
case r.URL.Path == "/api/service-accounts" && r.Method == http.MethodPost:
var req struct {
Name string `json:"name"`
Scope string `json:"scope"`
}
_ = json.NewDecoder(r.Body).Decode(&req)
if _, exists := f.accounts[req.Name]; exists {
writeJSON(w, http.StatusConflict, map[string]string{"error": "a service account with that name already exists"})
return
}
f.nextID++
id := f.nextID
f.accounts[req.Name] = id
f.keyMints[id] = 1
writeJSON(w, http.StatusCreated, tdclient.CreateServiceAccountResult{
ServiceAccount: tdclient.ServiceAccount{ID: id, Name: req.Name, Scope: req.Scope},
Key: tdclient.APIKey{ID: 1, Name: "initial", Key: fmt.Sprintf("instance-key-%d-initial", id)},
})
case r.URL.Path == "/api/service-accounts" && r.Method == http.MethodGet:
name := r.URL.Query().Get("name")
id, exists := f.accounts[name]
if !exists {
writeJSON(w, http.StatusOK, []tdclient.ServiceAccount{})
return
}
writeJSON(w, http.StatusOK, []tdclient.ServiceAccount{{ID: id, Name: name, Scope: "instance"}})
default:
if id, name, ok := parseKeysPath(r.URL.Path); ok && r.Method == http.MethodPost {
f.keyMints[id]++
writeJSON(w, http.StatusCreated, tdclient.APIKey{
ID: int64(f.keyMints[id]), Name: name,
Key: fmt.Sprintf("instance-key-%d-mint%d", id, f.keyMints[id]),
})
return
}
w.WriteHeader(http.StatusNotFound)
}
}
func parseKeysPath(path string) (id int64, mintName string, ok bool) {
var parsedID int64
n, err := fmt.Sscanf(path, "/api/service-accounts/%d/keys", &parsedID)
if err != nil || n != 1 {
return 0, "", false
}
return parsedID, "minted", true
}
func writeJSON(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(v)
}
var _ = Describe("TerdutServer Controller", func() {
@@ -63,10 +140,23 @@ var _ = Describe("TerdutServer Controller", func() {
AfterEach(func(ctx SpecContext) {
srv := &terdutv1alpha1.TerdutServer{}
if err := k8sClient.Get(ctx, objKey, srv); err == nil {
Expect(k8sClient.Delete(ctx, srv)).To(Succeed())
srv.Finalizers = nil
_ = k8sClient.Update(ctx, srv)
_ = k8sClient.Delete(ctx, srv)
}
for _, n := range []string{checkpointSecretNameFor(name), credentialsSecretNameFor(name)} {
_ = k8sClient.Delete(ctx, &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: n, Namespace: operatorNamespace}})
}
})
dsnSpec := func() terdutv1alpha1.TerdutServerSpec {
return terdutv1alpha1.TerdutServerSpec{
Image: terdutv1alpha1.ImageSpec{Repository: "example.invalid/terdut-server", Tag: "test"},
Networking: terdutv1alpha1.NetworkingSpec{Hostname: "terdut.example.invalid", ServicePort: 8080},
Database: terdutv1alpha1.DatabaseSpec{DSN: "postgres://terdut@test-postgres:5432/terdut?sslmode=require"},
}
}
createServer := func(ctx context.Context, spec terdutv1alpha1.TerdutServerSpec) {
srv := &terdutv1alpha1.TerdutServer{
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: operatorNamespace},
@@ -75,123 +165,235 @@ var _ = Describe("TerdutServer Controller", func() {
Expect(k8sClient.Create(ctx, srv)).To(Succeed())
}
reconcileOnce := func(ctx context.Context) {
_, err := reconciler.Reconcile(ctx, reconcile.Request{NamespacedName: objKey})
reconcileOnce := func(ctx context.Context) ctrl.Result {
res, err := reconciler.Reconcile(ctx, reconcile.Request{NamespacedName: objKey})
Expect(err).NotTo(HaveOccurred())
return res
}
markDeploymentReady := func(ctx context.Context) {
var deploy appsv1.Deployment
Expect(k8sClient.Get(ctx, objKey, &deploy)).To(Succeed())
deploy.Status.ReadyReplicas = 1
deploy.Status.Replicas = 1
Expect(k8sClient.Status().Update(ctx, &deploy)).To(Succeed())
}
readyCondition := func(ctx context.Context) metav1.Condition {
srv := &terdutv1alpha1.TerdutServer{}
Expect(k8sClient.Get(ctx, objKey, srv)).To(Succeed())
c := meta.FindStatusCondition(srv.Status.Conditions, terdutv1alpha1.ConditionReady)
Expect(c).NotTo(BeNil(), "Ready condition should always be set after a reconcile")
Expect(c).NotTo(BeNil(), "Ready condition should always be set after a reconcile past the finalizer-add pass")
return *c
}
When("spec.credentialsSecretRef is unset", func() {
It("reports Ready: False, reason CredentialsSecretRefRequired, and makes no API call", func(ctx SpecContext) {
createServer(ctx, terdutv1alpha1.TerdutServerSpec{Endpoint: unusedEndpoint})
Describe("bring-your-own DSN path", func() {
It("reaches Ready through the full lifecycle: finalizer, Deployment/Service, wait, bootstrap", func(ctx SpecContext) {
fake, fakeSrv := newFakeTerdutServer()
DeferCleanup(fakeSrv.Close)
reconciler.NewClient = func(string) *tdclient.Client { return tdclient.New(fakeSrv.URL) }
createServer(ctx, dsnSpec())
// Pass 1: adds the finalizer and returns early.
reconcileOnce(ctx)
srv := &terdutv1alpha1.TerdutServer{}
Expect(k8sClient.Get(ctx, objKey, srv)).To(Succeed())
Expect(srv.Finalizers).To(ContainElement(finalizerName))
cond := readyCondition(ctx)
Expect(cond.Status).To(Equal(metav1.ConditionFalse))
Expect(cond.Reason).To(Equal(terdutv1alpha1.ReasonCredentialsSecretRefRequired))
})
})
When("the referenced Secret does not exist", func() {
It("reports Ready: False, reason CredentialsSecretNotFound", func(ctx SpecContext) {
createServer(ctx, terdutv1alpha1.TerdutServerSpec{
Endpoint: unusedEndpoint,
CredentialsSecretRef: &terdutv1alpha1.SecretKeyRef{Name: "does-not-exist", Key: tokenKey},
})
// Pass 2: creates Deployment + Service, waits for readiness.
reconcileOnce(ctx)
Expect(readyCondition(ctx).Reason).To(Equal(terdutv1alpha1.ReasonWaitingForDeployment))
cond := readyCondition(ctx)
Expect(cond.Status).To(Equal(metav1.ConditionFalse))
Expect(cond.Reason).To(Equal(terdutv1alpha1.ReasonCredentialsSecretNotFound))
})
})
When("the referenced Secret exists but has no data under the given key", func() {
It("reports Ready: False, reason CredentialsSecretInvalid", func(ctx SpecContext) {
secret := &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{Name: name + "-creds", Namespace: operatorNamespace},
Data: map[string][]byte{"wrong-key": []byte("tdsa_something")},
var deploy appsv1.Deployment
Expect(k8sClient.Get(ctx, objKey, &deploy)).To(Succeed())
Expect(deploy.Spec.Template.Spec.Containers[0].Image).To(Equal("example.invalid/terdut-server:test"))
envNames := map[string]string{}
for _, e := range deploy.Spec.Template.Spec.Containers[0].Env {
envNames[e.Name] = e.Value
}
Expect(k8sClient.Create(ctx, secret)).To(Succeed())
defer func() { _ = k8sClient.Delete(ctx, secret) }()
Expect(envNames).To(HaveKeyWithValue("TERDUT_DB_DSN", "postgres://terdut@test-postgres:5432/terdut?sslmode=require"))
Expect(envNames).To(HaveKeyWithValue("TERDUT_OPERATOR_MODE", "true"))
createServer(ctx, terdutv1alpha1.TerdutServerSpec{
Endpoint: unusedEndpoint,
CredentialsSecretRef: &terdutv1alpha1.SecretKeyRef{Name: secret.Name, Key: tokenKey},
})
var svc corev1.Service
Expect(k8sClient.Get(ctx, objKey, &svc)).To(Succeed())
Expect(svc.Spec.Ports[0].Port).To(Equal(int32(8080)))
// Simulate the Deployment becoming ready (envtest has no
// kubelet/deployment-controller to do this for real).
markDeploymentReady(ctx)
// Pass 3: bootstraps for real against the fake server.
reconcileOnce(ctx)
cond := readyCondition(ctx)
Expect(cond.Status).To(Equal(metav1.ConditionFalse))
Expect(cond.Reason).To(Equal(terdutv1alpha1.ReasonCredentialsSecretInvalid))
Expect(fake.bootstrapped).To(BeTrue())
Expect(k8sClient.Get(ctx, objKey, srv)).To(Succeed())
ready := meta.FindStatusCondition(srv.Status.Conditions, terdutv1alpha1.ConditionReady)
Expect(ready.Status).To(Equal(metav1.ConditionTrue))
Expect(ready.Reason).To(Equal(terdutv1alpha1.ReasonAdopted))
Expect(srv.Status.CredentialsSecretRef).NotTo(BeNil())
Expect(srv.Status.CredentialsSecretRef.Key).To(Equal(credentialsSecretDataKey))
var credsSecret corev1.Secret
Expect(k8sClient.Get(ctx, types.NamespacedName{
Name: srv.Status.CredentialsSecretRef.Name, Namespace: operatorNamespace,
}, &credsSecret)).To(Succeed())
Expect(string(credsSecret.Data[credentialsSecretDataKey])).To(Equal("instance-key-1-initial"))
// The checkpoint is cleaned up once the lasting credential is
// written (DESIGN.md §6 point 2).
var checkpoint corev1.Secret
err := k8sClient.Get(ctx, types.NamespacedName{Name: checkpointSecretName(srv), Namespace: operatorNamespace}, &checkpoint)
Expect(err).To(HaveOccurred())
})
})
When("the Secret is valid but the server is unreachable", func() {
It("reports Ready: False, reason ServerUnreachable", func(ctx SpecContext) {
secret := &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{Name: name + "-creds", Namespace: operatorNamespace},
Data: map[string][]byte{tokenKey: []byte("tdsa_something")},
}
Expect(k8sClient.Create(ctx, secret)).To(Succeed())
defer func() { _ = k8sClient.Delete(ctx, secret) }()
Describe("the adopt-on-409 recovery path", func() {
It("mints a fresh key instead of erroring when the service account already exists", func(ctx SpecContext) {
fake, fakeSrv := newFakeTerdutServer()
DeferCleanup(fakeSrv.Close)
fake.bootstrapped = true // an earlier attempt already bootstrapped...
fake.nextID = 1
fake.accounts[serviceAccountName] = 1 // ...and already created the service account.
fake.keyMints[1] = 1
reconciler.NewClient = func(string) *tdclient.Client { return tdclient.New(fakeSrv.URL) }
createServer(ctx, terdutv1alpha1.TerdutServerSpec{
// Port 1 is never listening (closed cleanly and immediately,
// unlike an unroutable address which would hang on a
// connect timeout) -- keeps the test fast.
Endpoint: "http://127.0.0.1:1",
CredentialsSecretRef: &terdutv1alpha1.SecretKeyRef{Name: secret.Name, Key: tokenKey},
})
reconcileOnce(ctx)
createServer(ctx, dsnSpec())
reconcileOnce(ctx) // finalizer
reconcileOnce(ctx) // Deployment/Service, WaitingForDeployment
markDeploymentReady(ctx)
cond := readyCondition(ctx)
Expect(cond.Status).To(Equal(metav1.ConditionFalse))
Expect(cond.Reason).To(Equal(terdutv1alpha1.ReasonServerUnreachable))
})
})
// The earlier attempt's checkpoint survived (that's how this
// reconcile can authenticate at all to recover).
Expect(reconciler.writeSecret(ctx, checkpointSecretName(&terdutv1alpha1.TerdutServer{
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: operatorNamespace},
}), "admin-key-raw")).To(Succeed())
When("the Secret is valid and the server answers /api/version", func() {
It("reports Ready: True, Bootstrapped: True, and mirrors credentialsSecretRef into status", func(ctx SpecContext) {
fake := fakeTerdutServer("v0.20.0")
DeferCleanup(fake.Close)
secret := &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{Name: name + "-creds", Namespace: operatorNamespace},
Data: map[string][]byte{tokenKey: []byte("tdsa_something")},
}
Expect(k8sClient.Create(ctx, secret)).To(Succeed())
defer func() { _ = k8sClient.Delete(ctx, secret) }()
createServer(ctx, terdutv1alpha1.TerdutServerSpec{
Endpoint: fake.URL,
CredentialsSecretRef: &terdutv1alpha1.SecretKeyRef{Name: secret.Name, Key: tokenKey},
})
reconcileOnce(ctx)
srv := &terdutv1alpha1.TerdutServer{}
Expect(k8sClient.Get(ctx, objKey, srv)).To(Succeed())
Expect(meta.FindStatusCondition(srv.Status.Conditions, terdutv1alpha1.ConditionReady).Status).To(Equal(metav1.ConditionTrue))
ready := meta.FindStatusCondition(srv.Status.Conditions, terdutv1alpha1.ConditionReady)
Expect(ready).NotTo(BeNil())
Expect(ready.Status).To(Equal(metav1.ConditionTrue))
Expect(ready.Reason).To(Equal(terdutv1alpha1.ReasonAdopted))
var credsSecret corev1.Secret
Expect(k8sClient.Get(ctx, types.NamespacedName{
Name: srv.Status.CredentialsSecretRef.Name, Namespace: operatorNamespace,
}, &credsSecret)).To(Succeed())
// Minted fresh, not the (never-seen-by-this-reconcile) "initial"
// key from the account's original creation.
Expect(string(credsSecret.Data[credentialsSecretDataKey])).To(Equal("instance-key-1-mint2"))
})
})
bootstrapped := meta.FindStatusCondition(srv.Status.Conditions, terdutv1alpha1.ConditionBootstrapped)
Expect(bootstrapped).NotTo(BeNil())
Expect(bootstrapped.Status).To(Equal(metav1.ConditionTrue))
Describe("the BootstrapStateLost path", func() {
It("fails closed when the server reports already-bootstrapped with no checkpoint to recover from", func(ctx SpecContext) {
fake, fakeSrv := newFakeTerdutServer()
_ = fake
DeferCleanup(fakeSrv.Close)
fake.bootstrap403 = true
reconciler.NewClient = func(string) *tdclient.Client { return tdclient.New(fakeSrv.URL) }
Expect(srv.Status.CredentialsSecretRef).NotTo(BeNil())
Expect(srv.Status.CredentialsSecretRef.Name).To(Equal(secret.Name))
Expect(srv.Status.CredentialsSecretRef.Key).To(Equal(tokenKey))
Expect(srv.Status.ObservedGeneration).To(Equal(srv.Generation))
createServer(ctx, dsnSpec())
reconcileOnce(ctx) // finalizer
reconcileOnce(ctx) // Deployment/Service
markDeploymentReady(ctx)
reconcileOnce(ctx)
cond := readyCondition(ctx)
Expect(cond.Status).To(Equal(metav1.ConditionFalse))
Expect(cond.Reason).To(Equal(terdutv1alpha1.ReasonBootstrapStateLost))
})
})
Describe("the Zalando postgresClusterRef path", func() {
zalandoSpec := func(clusterName string) terdutv1alpha1.TerdutServerSpec {
s := dsnSpec()
s.Database = terdutv1alpha1.DatabaseSpec{
PostgresClusterRef: &terdutv1alpha1.PostgresClusterRef{Name: clusterName},
}
return s
}
It("waits with reason PostgresClusterNotFound when the postgresql CR doesn't exist yet", func(ctx SpecContext) {
createServer(ctx, zalandoSpec("missing-cluster"))
reconcileOnce(ctx) // finalizer
reconcileOnce(ctx)
Expect(readyCondition(ctx).Reason).To(Equal(terdutv1alpha1.ReasonPostgresClusterNotFound))
})
It("resolves a real DSN and reaches Ready once the CR and its generated Secret both exist", func(ctx SpecContext) {
clusterName := "pg-" + name
cluster := &unstructured.Unstructured{}
cluster.SetGroupVersionKind(postgresqlGVK)
cluster.SetName(clusterName)
cluster.SetNamespace(operatorNamespace)
Expect(unstructured.SetNestedField(cluster.Object, map[string]any{}, "spec")).To(Succeed())
Expect(k8sClient.Create(ctx, cluster)).To(Succeed())
DeferCleanup(func() { _ = k8sClient.Delete(ctx, cluster) })
credsSecretName := fmt.Sprintf("terdut.%s.credentials.postgresql.acid.zalan.do", clusterName)
zalandoSecret := &corev1.Secret{
ObjectMeta: metav1.ObjectMeta{Name: credsSecretName, Namespace: operatorNamespace},
Data: map[string][]byte{"password": []byte("whatever")},
}
Expect(k8sClient.Create(ctx, zalandoSecret)).To(Succeed())
DeferCleanup(func() { _ = k8sClient.Delete(ctx, zalandoSecret) })
fake, fakeSrv := newFakeTerdutServer()
_ = fake
DeferCleanup(fakeSrv.Close)
reconciler.NewClient = func(string) *tdclient.Client { return tdclient.New(fakeSrv.URL) }
createServer(ctx, zalandoSpec(clusterName))
reconcileOnce(ctx) // finalizer
reconcileOnce(ctx) // Deployment/Service created with resolved DB env
var deploy appsv1.Deployment
Expect(k8sClient.Get(ctx, objKey, &deploy)).To(Succeed())
var dsn string
for _, e := range deploy.Spec.Template.Spec.Containers[0].Env {
if e.Name == "TERDUT_DB_DSN" {
dsn = e.Value
}
}
Expect(dsn).To(Equal(fmt.Sprintf("postgres://terdut@%s.%s.svc:5432/terdut?sslmode=require", clusterName, operatorNamespace)))
markDeploymentReady(ctx)
reconcileOnce(ctx)
Expect(readyCondition(ctx).Status).To(Equal(metav1.ConditionTrue))
})
})
Describe("deletion", func() {
It("removes the credentials and checkpoint Secrets and the finalizer", func(ctx SpecContext) {
fake, fakeSrv := newFakeTerdutServer()
_ = fake
DeferCleanup(fakeSrv.Close)
reconciler.NewClient = func(string) *tdclient.Client { return tdclient.New(fakeSrv.URL) }
createServer(ctx, dsnSpec())
reconcileOnce(ctx)
reconcileOnce(ctx)
markDeploymentReady(ctx)
reconcileOnce(ctx)
srv := &terdutv1alpha1.TerdutServer{}
Expect(k8sClient.Get(ctx, objKey, srv)).To(Succeed())
credsName := srv.Status.CredentialsSecretRef.Name
Expect(k8sClient.Delete(ctx, srv)).To(Succeed())
reconcileOnce(ctx) // runs the finalizer
err := k8sClient.Get(ctx, objKey, srv)
Expect(err).To(HaveOccurred(), "the TerdutServer itself should be gone once the finalizer clears")
var leftover corev1.Secret
err = k8sClient.Get(ctx, types.NamespacedName{Name: credsName, Namespace: operatorNamespace}, &leftover)
Expect(err).To(HaveOccurred(), "the credentials Secret should have been cleaned up by the finalizer")
})
})
@@ -205,12 +407,12 @@ var _ = Describe("TerdutServer Controller", func() {
})
})
var _ = Describe("TerdutServerReconciler sanity", func() {
It("treats a real apierrors.IsNotFound the same as any other caller would", func() {
// Guards against a refactor accidentally swapping in a different
// not-found check that stops matching what client.Client actually
// returns.
Expect(apierrors.IsNotFound(apierrors.NewNotFound(
terdutv1alpha1.GroupVersion.WithResource("terdutservers").GroupResource(), "x"))).To(BeTrue())
})
})
// checkpointSecretNameFor/credentialsSecretNameFor let AfterEach clean up
// without needing a live TerdutServer object (it may already be gone by
// then in the deletion test).
func checkpointSecretNameFor(name string) string {
return fmt.Sprintf("default.%s-bootstrap-admin", name)
}
func credentialsSecretNameFor(name string) string {
return fmt.Sprintf("default.%s-instance-credentials", name)
}
@@ -0,0 +1,134 @@
package controller
import (
"context"
"fmt"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"sigs.k8s.io/controller-runtime/pkg/client"
terdutv1alpha1 "git.ryuvia.com/niklas/terdut-operator/api/v1alpha1"
)
// databaseError carries a condition reason/message out of resolveDatabaseEnv
// — distinct from a plain error, since "the database isn't ready yet" is an
// expected, requeue-and-retry outcome (DESIGN.md §8), not a reconcile
// failure.
type databaseError struct {
reason string
message string
}
func (e *databaseError) Error() string { return e.message }
// classifyClusterGetError turns a failed Get of the postgresql.acid.zalan.do
// CR into the right condition — distinct from a plain reconcile error, and
// its own function (not inlined) so it's unit-testable against a
// hand-constructed error without needing a real client at all: a
// *meta.NoKindMatchError means the CRD itself isn't installed
// (ReasonPostgresOperatorCRDNotInstalled, §8/§9's graceful-degradation
// case), apierrors.IsNotFound means the CRD exists but this particular
// object doesn't (yet) (ReasonPostgresClusterNotFound).
func classifyClusterGetError(err error, clusterName, namespace string) *databaseError {
switch {
case meta.IsNoMatchError(err):
return &databaseError{
reason: terdutv1alpha1.ReasonPostgresOperatorCRDNotInstalled,
message: "spec.database.postgresClusterRef is set but the postgresql.acid.zalan.do CRD isn't installed in this cluster",
}
case apierrors.IsNotFound(err):
return &databaseError{
reason: terdutv1alpha1.ReasonPostgresClusterNotFound,
message: fmt.Sprintf("postgresql.acid.zalan.do %q not found in namespace %q", clusterName, namespace),
}
default:
// Any other error (RBAC, transient API failure) is a real
// reconcile error, not a condition to report and wait out --
// callers still surface it as a databaseError so resolveDatabaseEnv
// has one return shape, but Reconcile treats every databaseError as
// a wait-and-retry outcome today. Revisit if that ever matters in
// practice (a persistent RBAC misconfiguration looks identical to
// "not ready yet" until someone reads the condition message).
return &databaseError{reason: terdutv1alpha1.ReasonPostgresClusterNotFound, message: err.Error()}
}
}
// resolveDatabaseEnv implements DESIGN.md §8's two Postgres paths. It never
// reads a password's value — only ever wires a secretKeyRef into the
// Deployment's env, the same way the chart does — so it needs no RBAC on
// Secret *data* for this, only on the postgresql.acid.zalan.do CR itself
// (the XValidation rule on TerdutServerSpec.Database guarantees exactly one
// of the two fields below is set, so there's no third case to handle).
func (r *TerdutServerReconciler) resolveDatabaseEnv(ctx context.Context, srv *terdutv1alpha1.TerdutServer) ([]corev1.EnvVar, *databaseError) {
db := srv.Spec.Database
if db.PostgresClusterRef != nil {
return r.resolveZalandoDatabaseEnv(ctx, srv, db.PostgresClusterRef.Name)
}
env := []corev1.EnvVar{{Name: "TERDUT_DB_DSN", Value: db.DSN}}
if db.PasswordSecretRef != nil {
env = append(env, corev1.EnvVar{
Name: "PGPASSWORD",
ValueFrom: &corev1.EnvVarSource{
SecretKeyRef: &corev1.SecretKeySelector{
LocalObjectReference: corev1.LocalObjectReference{Name: db.PasswordSecretRef.Name},
Key: db.PasswordSecretRef.Key,
},
},
})
}
return env, nil
}
// resolveZalandoDatabaseEnv resolves a Zalando postgres-operator
// `postgresql` CR into a DSN + PGPASSWORD secretKeyRef, without reading the
// generated credentials Secret's value — only confirming it exists, the
// same "exists, don't read" posture the DSN path takes toward its own
// password Secret.
//
// By convention (DESIGN.md §8, PostgresClusterRef's own doc comment) the
// database and role are both named "terdut", and the primary Service is
// named after the cluster CR itself — Zalando's own naming convention, not
// something its status exposes as a field to read.
func (r *TerdutServerReconciler) resolveZalandoDatabaseEnv(
ctx context.Context, srv *terdutv1alpha1.TerdutServer, clusterName string,
) ([]corev1.EnvVar, *databaseError) {
cluster := &unstructured.Unstructured{}
cluster.SetGroupVersionKind(postgresqlGVK)
if err := r.Get(ctx, client.ObjectKey{Namespace: srv.Namespace, Name: clusterName}, cluster); err != nil {
return nil, classifyClusterGetError(err, clusterName, srv.Namespace)
}
credsSecretName := fmt.Sprintf("terdut.%s.credentials.postgresql.acid.zalan.do", clusterName)
var secret corev1.Secret
if err := r.Get(ctx, client.ObjectKey{Namespace: srv.Namespace, Name: credsSecretName}, &secret); err != nil {
if apierrors.IsNotFound(err) {
return nil, &databaseError{
reason: terdutv1alpha1.ReasonPostgresClusterNotFound,
message: fmt.Sprintf(
"postgresql.acid.zalan.do %q found, but its generated credentials Secret %q doesn't exist yet",
clusterName, credsSecretName),
}
}
return nil, &databaseError{reason: terdutv1alpha1.ReasonPostgresClusterNotFound, message: err.Error()}
}
host := fmt.Sprintf("%s.%s.svc", clusterName, srv.Namespace)
dsn := fmt.Sprintf("postgres://terdut@%s:5432/terdut?sslmode=require", host)
return []corev1.EnvVar{
{Name: "TERDUT_DB_DSN", Value: dsn},
{
Name: "PGPASSWORD",
ValueFrom: &corev1.EnvVarSource{
SecretKeyRef: &corev1.SecretKeySelector{
LocalObjectReference: corev1.LocalObjectReference{Name: credsSecretName},
Key: "password",
},
},
},
}, nil
}
@@ -0,0 +1,47 @@
package controller
import (
"errors"
"testing"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
"k8s.io/apimachinery/pkg/runtime/schema"
terdutv1alpha1 "git.ryuvia.com/niklas/terdut-operator/api/v1alpha1"
)
// These exercise classifyClusterGetError directly, against hand-constructed
// errors, rather than through envtest -- the point is the classification
// logic, not a real Get, and the "CRD not installed" case specifically
// can't be produced in the shared envtest suite (its CRDs, stub included,
// are always installed for the whole suite's lifetime).
func TestClassifyClusterGetError_CRDNotInstalled(t *testing.T) {
noMatch := &meta.NoKindMatchError{GroupKind: schema.GroupKind{Group: "acid.zalan.do", Kind: "postgresql"}}
got := classifyClusterGetError(noMatch, "terdut-postgres", "oncall")
if got.reason != terdutv1alpha1.ReasonPostgresOperatorCRDNotInstalled {
t.Errorf("reason = %q, want %q", got.reason, terdutv1alpha1.ReasonPostgresOperatorCRDNotInstalled)
}
}
func TestClassifyClusterGetError_ClusterNotFound(t *testing.T) {
notFound := apierrors.NewNotFound(schema.GroupResource{Group: "acid.zalan.do", Resource: "postgresqls"}, "terdut-postgres")
got := classifyClusterGetError(notFound, "terdut-postgres", "oncall")
if got.reason != terdutv1alpha1.ReasonPostgresClusterNotFound {
t.Errorf("reason = %q, want %q", got.reason, terdutv1alpha1.ReasonPostgresClusterNotFound)
}
}
func TestClassifyClusterGetError_OtherErrorStillReturnsACondition(t *testing.T) {
// Not one of the two classified cases -- still returns a databaseError
// (Reconcile treats every databaseError as wait-and-retry today; see
// the function's own comment on why), never nil, so a caller can't
// accidentally treat an unclassified error as success.
got := classifyClusterGetError(errors.New("etcd is on fire"), "terdut-postgres", "oncall")
if got == nil {
t.Fatal("classifyClusterGetError returned nil for an unclassified error")
}
}
@@ -0,0 +1,219 @@
package controller
import (
"context"
"fmt"
"strings"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/intstr"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
terdutv1alpha1 "git.ryuvia.com/niklas/terdut-operator/api/v1alpha1"
)
// labelsFor returns the selector labels for srv's Deployment/Service —
// fixed once a Deployment exists (its selector is immutable), so this must
// never depend on anything in srv.Spec that could change later.
func labelsFor(srv *terdutv1alpha1.TerdutServer) map[string]string {
return map[string]string{
"app.kubernetes.io/name": "terdut-server",
"app.kubernetes.io/instance": srv.Name,
}
}
// reconcileDeployment creates or updates the Deployment running
// terdut-server, owned by srv (DESIGN.md §7: same-namespace generated
// objects carry a plain OwnerReference, no finalizer). Returns the live
// object (not just the desired one) so the caller can check
// status.readyReplicas.
func (r *TerdutServerReconciler) reconcileDeployment(
ctx context.Context, srv *terdutv1alpha1.TerdutServer, dbEnv []corev1.EnvVar,
) (*appsv1.Deployment, error) {
deploy := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: srv.Name, Namespace: srv.Namespace}}
_, err := controllerutil.CreateOrUpdate(ctx, r.Client, deploy, func() error {
replicas := srv.Spec.Replicas
if replicas == 0 {
replicas = 1
}
labels := labelsFor(srv)
deploy.Spec.Replicas = &replicas
deploy.Spec.Selector = &metav1.LabelSelector{MatchLabels: labels}
// Recreate, not RollingUpdate: the sweeper and the notifier are
// unsynchronised singletons inside terdut-server, and two replicas
// overlapping during a rollout would both page for the same
// incident (matches the chart's own deployment.yaml comment).
deploy.Spec.Strategy = appsv1.DeploymentStrategy{Type: appsv1.RecreateDeploymentStrategyType}
deploy.Spec.Template = corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{Labels: labels},
Spec: corev1.PodSpec{
EnableServiceLinks: new(false),
Containers: []corev1.Container{{
Name: "terdut-server",
Image: fmt.Sprintf("%s:%s", srv.Spec.Image.Repository, srv.Spec.Image.Tag),
Ports: []corev1.ContainerPort{{
Name: "http",
ContainerPort: servicePort(srv),
Protocol: corev1.ProtocolTCP,
}},
Env: buildEnv(srv, dbEnv),
LivenessProbe: healthzProbe(),
ReadinessProbe: healthzProbe(),
}},
},
}
return controllerutil.SetControllerReference(srv, deploy, r.Scheme)
})
if err != nil {
return nil, err
}
// CreateOrUpdate's own object is desired-state-only on a fresh create
// (no .status yet); re-fetch so the ready-replica check the caller does
// next sees the real, live object.
if err := r.Get(ctx, client.ObjectKeyFromObject(deploy), deploy); err != nil {
return nil, err
}
return deploy, nil
}
// reconcileService creates or updates the ClusterIP Service in front of the
// Deployment, owned by srv.
func (r *TerdutServerReconciler) reconcileService(ctx context.Context, srv *terdutv1alpha1.TerdutServer) error {
svc := &corev1.Service{ObjectMeta: metav1.ObjectMeta{Name: srv.Name, Namespace: srv.Namespace}}
_, err := controllerutil.CreateOrUpdate(ctx, r.Client, svc, func() error {
svc.Spec.Selector = labelsFor(srv)
svc.Spec.Ports = []corev1.ServicePort{{
Name: "http",
Port: servicePort(srv),
TargetPort: intstr.FromString("http"),
Protocol: corev1.ProtocolTCP,
}}
return controllerutil.SetControllerReference(srv, svc, r.Scheme)
})
return err
}
func servicePort(srv *terdutv1alpha1.TerdutServer) int32 {
if srv.Spec.Networking.ServicePort == 0 {
return 8080
}
return srv.Spec.Networking.ServicePort
}
func healthzProbe() *corev1.Probe {
return &corev1.Probe{
ProbeHandler: corev1.ProbeHandler{
HTTPGet: &corev1.HTTPGetAction{
Path: "/healthz",
Port: intstr.FromString("http"),
},
},
InitialDelaySeconds: 5,
}
}
// buildEnv mirrors charts/terdut-server's own deployment.yaml env block
// field-for-field (confirmed against that source, not reconstructed from
// DESIGN.md's illustrative YAML alone) — dbEnv (TERDUT_DB_DSN, optionally
// PGPASSWORD) comes from resolveDatabaseEnv, since which of §8's two paths
// produced it doesn't matter past this point.
func buildEnv(srv *terdutv1alpha1.TerdutServer, dbEnv []corev1.EnvVar) []corev1.EnvVar {
env := []corev1.EnvVar{{Name: "TERDUT_ADDR", Value: fmt.Sprintf(":%d", servicePort(srv))}}
env = append(env, dbEnv...)
sweeper := srv.Spec.Sweeper
env = append(env,
corev1.EnvVar{Name: "TERDUT_STALE_AFTER", Value: sweeper.StaleAfter},
corev1.EnvVar{Name: "TERDUT_ARCHIVE_AFTER", Value: sweeper.ArchiveAfter},
)
deadman := srv.Spec.Deadman
env = append(env,
corev1.EnvVar{Name: "TERDUT_DEADMAN_MATCHERS", Value: deadman.Matchers},
corev1.EnvVar{Name: "TERDUT_DEADMAN_TIMEOUT", Value: deadman.Timeout},
corev1.EnvVar{Name: "TERDUT_DEADMAN_SEVERITY", Value: deadman.Severity},
)
notify := srv.Spec.Notify
if notify.NtfyURL != "" {
env = append(env,
corev1.EnvVar{Name: "TERDUT_NTFY_URL", Value: notify.NtfyURL},
corev1.EnvVar{Name: "TERDUT_NTFY_FALLBACK_TOPIC", Value: notify.FallbackTopic},
corev1.EnvVar{Name: "TERDUT_NOTIFY_REPEAT", Value: notify.RepeatEvery},
)
if notify.TokenSecretRef != nil {
env = append(env, corev1.EnvVar{
Name: "TERDUT_NTFY_TOKEN",
ValueFrom: secretEnvSource(notify.TokenSecretRef),
})
}
}
publicURL := fmt.Sprintf("https://%s", srv.Spec.Networking.Hostname)
env = append(env,
corev1.EnvVar{Name: "TERDUT_PUBLIC_URL", Value: publicURL},
corev1.EnvVar{Name: "TERDUT_PASSWORD_LOGIN", Value: boolString(srv.Spec.PasswordLogin)},
// Always on, unlike the chart's own default-off operatorMode: every
// write this operator's own TerdutTeam/EscalationRule/etc.
// controllers make goes through a service account already, and the
// whole reason to run this operator is gitops-managed config, not a
// human editing an operator-managed install's teams/policies by
// hand (chart's values.yaml comment: "a statement that something
// like terdut-operator... owns this install's configuration from
// here on" -- which is unconditionally true for anything this
// operator creates).
corev1.EnvVar{Name: "TERDUT_OPERATOR_MODE", Value: "true"},
)
if oidc := srv.Spec.OIDC; oidc.Enabled {
env = append(env,
corev1.EnvVar{Name: "TERDUT_OIDC_ISSUER", Value: oidc.Issuer},
corev1.EnvVar{Name: "TERDUT_OIDC_CLIENT_ID", Value: oidc.ClientID},
corev1.EnvVar{Name: "TERDUT_OIDC_NAME", Value: oidc.Name},
corev1.EnvVar{Name: "TERDUT_OIDC_SCOPES", Value: oidc.Scopes},
// terdut-server's own defaults for the claims/trust-email knobs
// the chart exposes but DESIGN.md's spec doesn't (§4.1's doc
// comment on OIDCSpec) -- not configurable here, not an
// oversight.
corev1.EnvVar{Name: "TERDUT_OIDC_USERNAME_CLAIM", Value: "preferred_username"},
corev1.EnvVar{Name: "TERDUT_OIDC_EMAIL_CLAIM", Value: "email"},
corev1.EnvVar{Name: "TERDUT_OIDC_GROUPS_CLAIM", Value: "groups"},
corev1.EnvVar{Name: "TERDUT_OIDC_TRUST_EMAIL", Value: "false"},
corev1.EnvVar{Name: "TERDUT_OIDC_SESSION_MAX_AGE", Value: oidc.SessionMaxAge},
)
if oidc.ClientSecretRef != nil {
env = append(env, corev1.EnvVar{
Name: "TERDUT_OIDC_CLIENT_SECRET",
ValueFrom: secretEnvSource(oidc.ClientSecretRef),
})
}
if len(oidc.AllowedGroups) > 0 {
env = append(env, corev1.EnvVar{Name: "TERDUT_OIDC_ALLOWED_GROUPS", Value: strings.Join(oidc.AllowedGroups, ",")})
}
if oidc.AdminGroup != "" {
env = append(env, corev1.EnvVar{Name: "TERDUT_OIDC_ADMIN_GROUP", Value: oidc.AdminGroup})
}
}
return env
}
func secretEnvSource(ref *terdutv1alpha1.SecretKeyRef) *corev1.EnvVarSource {
return &corev1.EnvVarSource{
SecretKeyRef: &corev1.SecretKeySelector{
LocalObjectReference: corev1.LocalObjectReference{Name: ref.Name},
Key: ref.Key,
},
}
}
func boolString(b bool) string {
if b {
return "true"
}
return "false"
}
@@ -0,0 +1,30 @@
# A deliberately minimal, permissive stub of Zalando postgres-operator's
# `postgresql` CRD -- test-only, installed into envtest so suite_test.go can
# create fixture `postgresql.acid.zalan.do` objects without a real
# postgres-operator controller running. This is NOT the real upstream CRD
# (which has a much larger structural schema); it exists purely so this
# repo's own controller logic -- which only ever Gets this object by name
# and never inspects its spec/status fields beyond existence -- has
# something matching the real GVK to read. A real cluster installs the
# genuine CRD via the postgres-operator chart; this file is never applied
# there.
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: postgresqls.acid.zalan.do
spec:
group: acid.zalan.do
names:
kind: postgresql
listKind: postgresqlList
plural: postgresqls
singular: postgresql
scope: Namespaced
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
x-kubernetes-preserve-unknown-fields: true