Skip to content

Recover Retained Resources After a Failed Lab Deployment

  • Purpose: Give a partial deployment an explicit terminal state without assuming that incomplete means disposable.
  • Owner: Platform operations
  • Impact: May keep, archive, or destroy resources according to the operator's explicit choice.
  • Rollback strategy: Keep the environment running until the failure and recovery value are understood.

Pre-checks

Record the error block and run-record path printed by the failed command. Then inspect the environment:

ENV=<environment>
REF=<blueprint-ref>
hyops show env --env "$ENV"

Check whether useful lab data already exists. Do not choose unprotected destroy only because deployment did not reach its final health check.

Execute

When the failure handoff offers a choice:

  1. Keep the environment running when diagnosis or manual recovery is required.
  2. Preserve declared lab state, verify it, then destroy when useful state exists.
  3. Destroy without preservation only when the environment is confirmed disposable.

For a later protected cleanup:

hyops blueprint destroy \
  --env "$ENV" \
  --ref "$REF" \
  --execute \
  --archive-before-destroy

For a confirmed disposable environment:

hyops blueprint destroy \
  --env "$ENV" \
  --ref "$REF" \
  --execute \
  --skip-archive

Verify

Confirm the final blueprint status and ongoing cost estimate. Re-run hyops show env --env "$ENV" to identify any intentionally retained reusable dependency.

Records

Retain the failed deploy record, the selected disposition, the later destroy record, and any recovery manifest. Redact secrets and private addresses before sharing diagnostic output.

Rollback

If cleanup fails, retain the environment and use the latest destroy run record to identify the blocked step. Correct that condition before retrying.

References