Cleanup the PostgreSQL App-Data DR Validation Lanes

• Purpose: Tear down the temporary validation clusters created by the PostgreSQL app-data DR drill without damaging the live primary lane, the source drill lane, shared DNS, or shared networking state. Owner: Platform engineering / SRE

• Trigger: Successful completion of the validation drill, failed validation run that must be reset, or cost-control teardown after acceptance.

• Impact: Deletes the isolated GCP validation VMs and isolated on-prem failback validation VMs, plus their attached HyOps module state lanes.

• Severity: P3 Pre-reqs: The validation drill logs and state have already been captured, and operators have confirmed no one still needs the temporary validation clusters.

• Rollback strategy: None for the validation lanes themselves. If teardown is accidental, rerun the validation drill to recreate them.

Context

This cleanup runbook is intentionally narrow.

It destroys only these validation lanes:

• platform/gcp/platform-vm#gcp_pg_vms_app_validation
• platform/postgresql-ha#postgresql_restore_gcp_app_validation
• platform/postgresql-ha-backup#postgresql_backup_config_gcp_app_validation
• platform/onprem/platform-vm#postgres_ha_vms_app_validation_drill
• platform/postgresql-ha#postgresql_restore_onprem_app_validation_drill
• platform/postgresql-ha-backup#postgresql_backup_config_onprem_app_validation_drill

It does not destroy:

• the live primary dev PostgreSQL lane
• the source drill lane at 10.12.0.41
• shared networking state such as org/gcp/wan-cloud-nat#gcp_pg_egress
• shared PowerDNS, NetBox, WAN, or runner state

Preconditions and safety checks

1. Confirm the validation outputs are already captured.

Recommended minimum:

• verification shell transcripts
• validation state JSON files

• relevant runner and module logs under ~/.hybridops/envs/dev/logs/ and ~/.hybridops/envs/drill/logs/

• Confirm you are destroying only the validation lanes.

Check the exact inputs files:

ls -1 "$HOME/.hybridops/envs/dev/config/modules/platform__gcp__platform-vm/instances/gcp_pg_vms_app_validation.inputs.yml"
ls -1 "$HOME/.hybridops/envs/dev/config/modules/platform__postgresql-ha/instances/postgresql_restore_gcp_app_validation.inputs.yml"
ls -1 "$HOME/.hybridops/envs/dev/config/modules/platform__postgresql-ha-backup/instances/postgresql_backup_config_gcp_app_validation.inputs.yml"

ls -1 "$HOME/.hybridops/envs/drill/config/modules/platform__onprem__platform-vm/instances/postgres_ha_vms_app_validation_drill.inputs.yml"
ls -1 "$HOME/.hybridops/envs/drill/config/modules/platform__postgresql-ha/instances/postgresql_restore_onprem_app_validation_drill.inputs.yml"
ls -1 "$HOME/.hybridops/envs/drill/config/modules/platform__postgresql-ha-backup/instances/postgresql_backup_config_onprem_app_validation_drill.inputs.yml"

1. Confirm the source drill lane still points at the original source leader.

   jq '.outputs.db_host' \
     "$HOME/.hybridops/envs/drill/state/modules/platform__postgresql-ha/latest.json"
   

Expected result:

• 10.12.0.41

Steps

1. Destroy the on-prem failback validation backup config

   cd /home/user/hybridops-tech/hybridops-core
   
   ./.venv/bin/hyops destroy --env drill \
     --module platform/postgresql-ha-backup \
     --state-instance postgresql_backup_config_onprem_app_validation_drill \
     --inputs "$HOME/.hybridops/envs/drill/config/modules/platform__postgresql-ha-backup/instances/postgresql_backup_config_onprem_app_validation_drill.inputs.yml"
   

Expected result:

• backup-config validation lane is removed

• Destroy the on-prem failback validation restore lane

  cd /home/user/hybridops-tech/hybridops-core
  
  ./.venv/bin/hyops destroy --env drill \
    --module platform/postgresql-ha \
    --state-instance postgresql_restore_onprem_app_validation_drill \
    --inputs "$HOME/.hybridops/envs/drill/config/modules/platform__postgresql-ha/instances/postgresql_restore_onprem_app_validation_drill.inputs.yml"
  

Expected result:

• on-prem validation PostgreSQL HA state is removed

• Destroy the on-prem failback validation VMs

  cd /home/user/hybridops-tech/hybridops-core
  
  ./.venv/bin/hyops destroy --env drill \
    --module platform/onprem/platform-vm \
    --state-instance postgres_ha_vms_app_validation_drill \
    --inputs "$HOME/.hybridops/envs/drill/config/modules/platform__onprem__platform-vm/instances/postgres_ha_vms_app_validation_drill.inputs.yml"
  

Expected result:

• pgfb-01, pgfb-02, and pgfb-03 are removed

• Destroy the GCP validation backup config lane

  cd /home/user/hybridops-tech/hybridops-core
  
  ./.venv/bin/hyops destroy --env dev \
    --module platform/postgresql-ha-backup \
    --state-instance postgresql_backup_config_gcp_app_validation \
    --inputs "$HOME/.hybridops/envs/dev/config/modules/platform__postgresql-ha-backup/instances/postgresql_backup_config_gcp_app_validation.inputs.yml"
  

Expected result:

• GCP backup-config validation lane is removed

• Destroy the GCP validation restore lane

  cd /home/user/hybridops-tech/hybridops-core
  
  ./.venv/bin/hyops destroy --env dev \
    --module platform/postgresql-ha \
    --state-instance postgresql_restore_gcp_app_validation \
    --inputs "$HOME/.hybridops/envs/dev/config/modules/platform__postgresql-ha/instances/postgresql_restore_gcp_app_validation.inputs.yml"
  

Expected result:

• GCP validation PostgreSQL HA state is removed

• Destroy the GCP validation VMs

  cd /home/user/hybridops-tech/hybridops-core
  
  ./.venv/bin/hyops destroy --env dev \
    --module platform/gcp/platform-vm \
    --state-instance gcp_pg_vms_app_validation \
    --inputs "$HOME/.hybridops/envs/dev/config/modules/platform__gcp__platform-vm/instances/gcp_pg_vms_app_validation.inputs.yml"
  

Expected result:

• platform-validation-pgapp-01, platform-validation-pgapp-02, and platform-validation-pgapp-03 are removed

• Leave the backup-run validation record in place unless you explicitly want to clear audit state

The state instance platform/postgresql-ha-backup#postgresql_backup_run_gcp_app_validation is an execution record, not a standing infrastructure lane.

Default guidance:

• keep it as an audit anchor for the drill
• remove it only if you intentionally want a clean validation-state slate

Verification

Confirm the validation states are gone or no longer ok:

ls -1 "$HOME/.hybridops/envs/dev/state/modules/platform__gcp__platform-vm/instances/" | rg 'gcp_pg_vms_app_validation' || true
ls -1 "$HOME/.hybridops/envs/dev/state/modules/platform__postgresql-ha/instances/" | rg 'postgresql_restore_gcp_app_validation' || true
ls -1 "$HOME/.hybridops/envs/drill/state/modules/platform__onprem__platform-vm/instances/" | rg 'postgres_ha_vms_app_validation_drill' || true
ls -1 "$HOME/.hybridops/envs/drill/state/modules/platform__postgresql-ha/instances/" | rg 'postgresql_restore_onprem_app_validation_drill' || true

Also confirm the source drill lane is still intact:

jq '.status, .outputs.db_host' \
  "$HOME/.hybridops/envs/drill/state/modules/platform__postgresql-ha/latest.json"

Expected result:

• source drill lane remains ok
• source drill leader remains 10.12.0.41

Post-actions and clean-up

• Keep the validation SQL files under ~/.hybridops/envs/drill/config/drvalidation/.
• Preserve accepted drill logs and state JSONs in your review artefacts before pruning runtime history.
• Do not destroy gcp_pg_egress from this cleanup unless you have separately verified that no other GCP validation or DR flows depend on it.

Related

Related reading

• Repeatable PostgreSQL App-Data DR Validation Drill
• Failover PostgreSQL HA to GCP (HyOps Blueprint)
• Failback PostgreSQL HA to On-Prem (HyOps Blueprint)
• PostgreSQL HA DR Cycle

References

• ~/.hybridops/envs/dev/config/modules/platform__gcp__platform-vm/instances/gcp_pg_vms_app_validation.inputs.yml
• ~/.hybridops/envs/dev/config/modules/platform__postgresql-ha/instances/postgresql_restore_gcp_app_validation.inputs.yml
• ~/.hybridops/envs/dev/config/modules/platform__postgresql-ha-backup/instances/postgresql_backup_config_gcp_app_validation.inputs.yml
• ~/.hybridops/envs/drill/config/modules/platform__onprem__platform-vm/instances/postgres_ha_vms_app_validation_drill.inputs.yml
• ~/.hybridops/envs/drill/config/modules/platform__postgresql-ha/instances/postgresql_restore_onprem_app_validation_drill.inputs.yml
• ~/.hybridops/envs/drill/config/modules/platform__postgresql-ha-backup/instances/postgresql_backup_config_onprem_app_validation_drill.inputs.yml

License: MIT-0 for code, CC-BY-4.0 for documentation