Imperative automation runs a sequence once. A reconciliation loop keeps asking: what differs between declared intent and reality now? It then makes one correction and asks again.

Because the loop is level-triggered, correctness depends on current state rather than seeing every event. After a restart, a missed notification, or a half-completed action, remaining state becomes the next input to the loop—not a special recovery path.

A declarative state store supplies desired state to a controller loop. The controller observes managed resources, compares desired and current state, applies idempotent corrections, reports status, and repeats until the states converge.

How It Works

  • Store desired state durably and keep controller-written status separate from owner-written intent.
  • Reconcile on change notifications and periodic resynchronization, so a missed event delays correction but cannot lose it permanently.
  • Compute the smallest correction from current state; make it idempotent.
  • Assign each field or resource to one controller.
  • Report conditions and retry with bounded backoff until reality matches intent.

Failure Modes

  • Two controllers own the same field and overwrite each other; the resource oscillates and both loops consume their retry budgets.
  • A stale observation triggers a correction against superseded state; newer intent is temporarily rolled back.
  • An edge-triggered controller misses an event and never revisits the resource, leaving permanent drift.
  • Invalid or unreachable desired state causes unbounded retry attempts at maximum backoff, saturating control-plane capacity while status stays unready.

Verification

  • Replica-controller convergence SLO: with target N, deleting one instance returns the observed count to N within 60 s.
  • Idempotency check: reconciling an already-converged resource twice produces zero external writes and no status change.
  • Crash check: stop the controller after its first side effect; after restart it converges without duplicate resources or skipped steps.
  • Steady-state signal: failed reconciliations stay below 1% per 15-minute window; every failure exposes a reason and next retry time.
  • The MAPE-K loop separates monitoring, analysis, planning, and execution when adaptation needs richer decisions than a direct state diff.
  • GitOps uses a version-controlled declaration as desired state and a reconciliation loop as its enforcement mechanism.
  • A watchdog checks liveness and usually restarts. A reconciler checks full state and can create, update, delete, or repair resources.

References