Skip to content

Decision Dispatcher Lifecycle (HyOps Module)

  • Purpose: Install the decision dispatcher that watches emitted decision records and writes normalized dispatch requests for downstream approval and execution.
  • Trigger: Decision service is active and the control plane needs a separate handoff stage before any DR or burst workflow can run.
  • Impact: Installs a systemd service on the control host, watches the decision-service records directory, and writes approval-gated request files under the dispatcher state directory.
  • Severity: P1
  • Rollback strategy: Destroy the module state instance to remove the dispatcher service and its runtime directory.

Context

Module ref: platform/network/decision-dispatcher

This module exists to keep the control loop clean:

  1. platform/network/decision-service evaluates signals and emits a decision record.
  2. platform/network/decision-dispatcher reads that record and turns it into a dispatch request.
  3. a later consumer or operator approval path decides whether to execute the request.

Current execution mode is intentionally conservative:

  • record-only

In this mode, the dispatcher does not run hyops itself. It only writes structured request files that can be reviewed or consumed by a later execution component.

Runtime paths on the control host:

  • decision records: /opt/hybridops/decision-service/state/records
  • dispatcher config: /opt/hybridops/decision-dispatcher/config/config.json
  • dispatcher state: /opt/hybridops/decision-dispatcher/state/state.json
  • dispatch requests: /opt/hybridops/decision-dispatcher/state/requests/
  • dispatcher log: /opt/hybridops/decision-dispatcher/logs/dispatcher.log

Preconditions

  • the control host is reachable through the normal inventory/SSH contract
  • platform/network/decision-service is already installed and healthy
  • dispatcher_routes is defined and maps decision types to target actions
  • route entries declare:
  • target_kind
  • target_ref
  • target_env
  • execution_plane

Example route shape:

dispatcher_routes:
  cutover:
    target_kind: blueprint
    target_ref: dr/postgresql-ha-failover-gcp@v1
    target_env: dev
    execution_plane: runner-local
    requires_approval: true
  failback:
    target_kind: blueprint
    target_ref: dr/postgresql-ha-failback-onprem@v1
    target_env: dev
    execution_plane: runner-local
    requires_approval: true

Execute

hyops apply --env dev \
  --module platform/network/decision-dispatcher \
  --inputs "$HOME/.hybridops/envs/dev/config/modules/platform__network__decision-dispatcher/latest.inputs.yml"

Verification

Check module outputs:

jq '.status, .outputs' \
  "$HOME/.hybridops/envs/dev/state/modules/platform__network__decision-dispatcher/latest.json"

Check live service state:

ssh -i "$HOME/.ssh/id_ed25519" opsadmin@5.161.116.216 \
  'sudo cat /opt/hybridops/decision-dispatcher/state/state.json'

Success indicators:

  • module state is status=ok
  • cap.control.decision_dispatcher = ready
  • decision_dispatcher.status = ready
  • decision_dispatcher.execution_mode = record-only
  • the hyops-decision-dispatcher service is active

Synthetic handoff validation

Use a disposable decision record to prove the control-loop handoff without executing any workflow:

  1. write a synthetic decision record into the decision-service records directory
  2. wait one dispatcher poll cycle
  3. confirm a request file appears under state/requests/
  4. confirm the request is awaiting-approval
  5. remove the synthetic record and request after verification

Example validation record:

{
  "decision_id": "test-cutover-001",
  "decision_type": "cutover",
  "rationale": "synthetic dispatcher handoff validation",
  "checks": [
    {"name": "gcp_billing", "status": "degraded"},
    {"name": "wan_state", "status": "degraded"}
  ]
}

Observed live validation on the current dev control host:

  • source record: test-cutover-001
  • emitted dispatch id: dispatch-20260313T041204Z-cutover-b8239f1f
  • status: awaiting-approval
  • target: dr/postgresql-ha-failover-gcp@v1
  • execution plane: runner-local

This validation was executed and then cleaned back out of the live host so no fake pending request remained behind.

Destroy

hyops destroy --env dev \
  --module platform/network/decision-dispatcher \
  --inputs "$HOME/.hybridops/envs/dev/config/modules/platform__network__decision-dispatcher/latest.inputs.yml"

Notes

  • record-only is the correct default. It keeps signal evaluation separate from execution.
  • The dispatcher does not replace runner execution. It prepares the request that a runner or future consumer will act on.
  • The dispatcher should remain on the shared control host alongside the decision service, DNS authority, and related control-plane services.
  • Approval remains explicit at this stage. Auto-execution belongs in a later consumer layer, not in the dispatcher itself.

References