Skip to content

Deploy Edge Control Plane (HyOps Blueprint)

Purpose: Deploy a governed edge control plane for WAN, observability, and cutover decisions.
Owner: Network/platform engineering
Trigger: New environment bring-up, edge rebuild, DR/burst rehearsal
Impact: Creates edge foundation and connects cloud/on-prem routing control-plane
Severity: P2
Pre-reqs: hyops init hetzner and hyops init gcp completed for target env, vault decrypt working, required secrets set.
Rollback strategy: Destroy module states in reverse dependency order or rerun with corrected inputs and skip_if_state_ok behavior.

Context

Blueprint ref: networking/edge-control-plane@v1
Location: hybridops-core/blueprints/networking/edge-control-plane@v1/blueprint.yml

Step flow:

  1. core/hetzner/vyos-image-seed
  2. org/hetzner/vyos-edge-foundation
  3. org/gcp/wan-hub-network
  4. org/gcp/wan-cloud-router
  5. org/gcp/wan-vpn-to-edge
  6. platform/network/wan-edge
  7. platform/network/edge-observability
  8. platform/network/dns-routing
  9. platform/network/decision-service

Target-state role split:

  • routed edge default: VyOS
  • shared control host: Linux
  • GCP routing hub: Cloud Router + NCC

Current DNS model:

  • platform/network/dns-routing remains the DNS cutover/update layer
  • first-class internal DNS authority target is now provider: powerdns-api
  • manual-command remains available as a fallback adapter
  • the recommended internal DNS authority topology is:
  • PowerDNS primary in the shared Hetzner / WAN-edge control plane
  • PowerDNS secondary on-prem for local resolution resilience

Important boundary:

  • this blueprint currently prepares the shared control plane around WAN edge, observability, and cutover logic
  • the first executable DNS authority path now lives in:
  • networking/powerdns-shared-primary@v1
  • networking/powerdns-onprem-secondary@v1
  • the first implementation intentionally colocates:
  • the writable primary on edge01
  • the read-only secondary on the on-prem runner host
  • this keeps cost down while preserving the clean authority/cutover separation
  • this aligns with the routed topology in the Network routing contract
  • the current platform/network/wan-edge step is therefore a compatibility path for Linux-edge labs, not the default routed-edge product posture

Current Hetzner image model:

  • the blueprint now uses core/hetzner/vyos-image-seed as the default state-first image path
  • if a matching Hetzner snapshot already exists, the step skips seeding and publishes state
  • if no matching snapshot exists, the step can seed one by using hcloud-upload-image
  • if the operator supplies a qcow2 URL, HyOps can now auto-wrap it for Hetzner image seeding, provided the execution host has a publicly reachable base URL configured
  • core/hetzner/vyos-image-register remains available as a compatibility path for externally managed images

Preconditions and safety checks

  1. Validate init readiness:
hyops init status --env dev
  1. Ensure required runtime secrets exist:
hyops secrets ensure --env dev WAN_IPSEC_PSK
hyops secrets ensure --env dev EDGE_OBS_GRAFANA_ADMIN_PASSWORD

Baseline profile note: - This blueprint keeps edge_observability in bootstrap mode (edge_obs_enable_receive=false, edge_obs_enable_store_gateway=false, edge_obs_enable_ruler=false), so object-store secret wiring is not required for first-pass E2E.

  1. Validate blueprint definition and preflight gate:
hyops blueprint validate --ref networking/edge-control-plane@v1
hyops blueprint preflight --env dev --ref networking/edge-control-plane@v1

Steps

  1. Execute blueprint
hyops blueprint deploy --env dev \
  --ref networking/edge-control-plane@v1 \
  --execute

  1. Track evidence while running

  2. HyOps prints the active step and evidence directory.

  3. Module logs are written under:

  4. ~/.hybridops/envs/<env>/logs/module/<module_ref_sanitized>/<run_id>/

  5. Verify final state

hyops state show --env dev --module org/hetzner/vyos-edge-foundation
hyops state show --env dev --module org/gcp/wan-vpn-to-edge
hyops state show --env dev --module platform/network/wan-edge
hyops state show --env dev --module platform/network/edge-observability
hyops state show --env dev --module platform/network/decision-service

Lifecycle test pattern

Use this sequence to validate module destroy/reapply behavior for operations phase:

# destroy in reverse dependency order
hyops destroy --env dev --module platform/network/decision-service --inputs /tmp/hyops-decision-service.dev.yml
hyops destroy --env dev --module platform/network/dns-routing --inputs /tmp/hyops-dns-routing.dev.yml
hyops destroy --env dev --module platform/network/edge-observability --inputs /tmp/hyops-edge-observability.dev.yml
hyops destroy --env dev --module platform/network/wan-edge --inputs /tmp/hyops-wan-edge-config.dev.yml

# re-apply via blueprint
hyops blueprint deploy --env dev --ref networking/edge-control-plane@v1 --execute

For destroy inputs, set explicit absent state where applicable:

  • dns_state: absent for platform/network/dns-routing
  • edge_obs_state: absent for platform/network/edge-observability

Advanced observability mode

To validate receive/store-gateway/ruler mode (with object-store), apply platform/network/edge-observability with explicit object-store config:

inventory_state_ref: org/hetzner/vyos-edge-foundation
inventory_vm_groups:
  edge:
    - edge01
edge_obs_enable_receive: true
edge_obs_enable_store_gateway: true
edge_obs_enable_ruler: true
edge_obs_hashring_endpoints:
  - 127.0.0.1:10907
edge_obs_objstore_config: |
  type: FILESYSTEM
  config:
    directory: /opt/hybridops/edge-observability/data/objstore
edge_obs_grafana_admin_password: "ChangeMe-Observability-Strong1!"

hyops preflight --env dev --strict --module platform/network/edge-observability --inputs /tmp/hyops-edge-observability.advanced.yml
hyops apply --env dev --module platform/network/edge-observability --inputs /tmp/hyops-edge-observability.advanced.yml

Verification

Success indicators:

  • Blueprint summary ends with status=ok.
  • All required steps report status=ok or skipped with valid prior state.
  • platform/network/decision-service state is ok and policy/action inputs are rendered.
  • if DNS automation is enabled, platform/network/dns-routing state records the desired target and provider (manual-command or powerdns-api)

Troubleshooting

  • Hetzner token invalid: rerun init and replace token.
hyops init hetzner --env dev --force
  • inputs.ssh_public_key contains placeholder: set a real public key before first Hetzner foundation apply.
  • missing required env var: WAN_IPSEC_PSK: set secret with hyops secrets ensure and rerun.
  • If you enable receive/store-gateway/ruler in edge_observability, also provide object-store config and include EDGE_OBS_OBJSTORE_CONFIG in required env.
  • For step-by-step debugging, run affected modules directly with hyops preflight/apply --module ... and inspect evidence path shown by CLI.
  • For internal DNS cutover, prefer the PowerDNS API path and the module example:
  • $HYOPS_CORE_ROOT/modules/platform/network/dns-routing/examples/inputs.powerdns.yml

References