Skip to content

Secrets lifecycle and responsibilities

Defines how secrets are handled from bootstrap through steady state and disaster recovery.

Design references:

Goals

  • Single source of truth for steady-state platform and application secret values.
  • Separation between bootstrap connectivity, steady-state runtime secrets, and break-glass recovery.
  • No plaintext secret values committed to Git (encrypted artefacts permitted).
  • No hardcoded credentials in playbooks, pipelines, templates, or images.

Runtime root and environments

HyOps stores local runtime artefacts under a runtime root (<root>), including credentials and the encrypted vault bundle.

<root> resolves to:

  • ~/.hybridops/envs/<env> when you pass --env <env> (recommended)
  • ~/.hybridops when no environment is specified (legacy single-env mode)

Common paths used by this guide:

  • Config: <root>/config/
  • Credentials: <root>/credentials/
  • Vault bundle: <root>/vault/bootstrap.vault.env

Reference:

Secret classes

Bootstrap connectivity secrets

Secrets used to allow operators and automation runners to authenticate to infrastructure control planes.

Primary artefacts (generated by hyops init <target>):

  • Config template: <root>/config/<target>.conf
  • Runtime credentials: <root>/credentials/<target>.credentials.tfvars (local-only, mode 0600)
  • Readiness marker: <root>/meta/<target>.ready.json (non-secret)

Constraints:

  • Not committed to Git.
  • Minimal scope: endpoints and credentials required to reach provider APIs.
  • Treated as sensitive local state (mode 0600).

Encrypted vault bundle

Encrypted vault bundle used to support non-interactive automation for bootstrap, CI, and DR.

  • File: <root>/vault/bootstrap.vault.env (ansible-vault encrypted env-format file)
  • Scope: bootstrap-only secrets and early pipeline credentials where required.
  • Vault password: stored out-of-band and injected at runtime (for example via CI secret or a local password provider).

Local workstation provider (recommended):

  • hyops vault bootstrap stores the vault password in pass (default entry: hybridops/ansible-vault).
  • The provider script is tools/secrets/vault/vault-pass.sh in the HybridOps.Core release root. It is discovered automatically when you run HyOps from the core repo, or can be overridden via hyops vault --script <path>.
  • After successful bootstrap, HyOps persists the default provider command at ~/.hybridops/config/vault-password-command so other HyOps commands can decrypt vault files without extra flags.
  • In a fresh shell, you may need to unlock your GPG key once before non-interactive operations succeed:
  • hyops vault password >/dev/null
  • If a command reports vault password command timed out while waiting for unlock, unlock once and optionally increase timeout:
  • hyops vault password >/dev/null
  • export HYOPS_VAULT_PASSWORD_COMMAND_TIMEOUT_S=300

Seeding secrets into the vault bundle:

  • Generate random defaults (bootstrap/labs/CI/DR):
  • hyops secrets ensure --env <env> KEY1 KEY2 ...
  • hyops secrets ensure --env <env> --module <module_ref> --inputs <inputs.yml>
  • Rotate existing keys: hyops secrets ensure --env <env> --force KEY1 [KEY2 ...]
  • Manual set (explicit overrides):
  • hyops secrets set --env <env> KEY=VALUE ...
  • hyops secrets set --env <env> --from-env KEY1 KEY2 ...
  • Enterprise sync (AKV -> vault bundle):
  • hyops secrets akv-sync --env <env> --vault-name <name> [--scope <scope>]

Notes:

  • The vault bundle is not a steady-state secrets store. It is a bootstrap/CI/DR convenience.
  • HyOps module state MUST NOT publish secret values. Modules publish env-var references (for example db_password_env) instead.

Constraints:

  • Plaintext derivatives must not be committed and must not persist beyond a single command execution.
  • The bundle is not a steady-state secret store.

Validation and failure semantics (fail-fast)

HyOps validates secrets in two layers:

  1. Module input validation (static)
  2. Modules declare which env keys they require via inputs.required_env.
  3. Any *_env fields (for example db_password_env, secret_key_env) MUST reference keys included in inputs.required_env.

  4. Validators fail before any provisioning work is executed if the contract is violated.

  5. Driver validation (runtime)

  6. Drivers validate that required env vars exist with non-empty values.
  7. The Ansible driver can optionally load <root>/vault/bootstrap.vault.env (when inputs.load_vault_env=true or when required keys are missing in the shell env).
  8. If required keys are still missing after vault merge, the command fails before running Ansible.

Recommended workflow:

  • Gate every config module run with:
  • hyops preflight --env <env> --strict --module <module_ref> --inputs <inputs.yml>

Typical failure messages are explicit:

  • Missing vault password provider:
  • [vault-pass] ERROR: entry not ready (run --bootstrap)
  • Missing required env vars:
  • missing required env vars: ... Generate random defaults: hyops secrets ensure --env <env> ... Or set explicit values: hyops secrets set --env <env> KEY=VALUE ...

Platform and application secrets

Secrets used by platform services and workloads.

Source of truth: Azure Key Vault (AKV).

Access patterns:

  • Platform tooling reads secrets from AKV via controlled access paths.
  • Kubernetes consumes secrets via External Secrets Operator syncing from AKV into Kubernetes Secret objects.

Constraints:

  • Git stores references and wiring, not secret values.
  • Kubernetes Secret objects are runtime projections and not primary storage.

Optional break-glass artefact

Optional DR artefact used only when required by DR policy or when the vault bundle scope is intentionally insufficient for recovery.

Example location:

  • docs/secrets/secrets.dr.enc.yaml

Constraints:

  • Minimal set required for the intended break-glass scenario.
  • Encrypted with SOPS.
  • Used only under a documented DR runbook.

Lifecycle by phase

Bootstrap

  1. Install the runtime and prerequisites:
  2. hyops setup all --sudo
  3. Bootstrap the vault password provider (workstations) or provide a CI password source:
  4. Workstation: hyops vault bootstrap
  5. Validate/unlock in current shell: hyops vault status-verbose then hyops vault password >/dev/null if needed
  6. CI: provide an Ansible Vault password source (for example ANSIBLE_VAULT_PASSWORD_FILE or --vault-password-command)
  7. Initialise provider access:
  8. hyops init <target> writes bootstrap config/credentials under <root>/config/ and <root>/credentials/
  9. Seed bootstrap-only secrets if required:
  10. Generate defaults: hyops secrets ensure --env <env> KEY1 KEY2 ...
  11. Override explicitly: hyops secrets set --env <env> KEY=VALUE ...
  12. Enterprise sync (AKV -> vault bundle): hyops secrets akv-sync --env <env> ...
  13. Use bootstrap connectivity for early provisioning:
  14. create foundations, establish AKV, and seed initial platform secrets.

Security intent:

  • Bootstrap artefacts remain minimal and local-only.
  • Secret values move to AKV as early as practical.

Steady state

  • AKV is authoritative for platform and application secret values.
  • External Secrets Operator projects AKV secrets into Kubernetes.
  • Bootstrap connectivity artefacts remain limited to infrastructure access and are not used as a general secret store.

Disaster recovery

  • DR automation is executed from an external runner using the encrypted vault bundle for non-interactive secret injection.
  • Typical actions: provision or scale target clusters, promote database replicas, perform DNS cutover.
  • Optional break-glass artefact is reserved for explicitly documented scenarios.

Design principles

  • Single active source of truth for steady-state secret values: AKV.
  • Git stores references, not secret values.
  • Encrypted vault bundles are permitted for bootstrap, CI, and DR only.
  • Bootstrap artefacts must not become a second live secrets system.
  • Rotation occurs at the source of truth (AKV) with controlled consumers.
  • Cloud backup identities must avoid key material in Terraform state.
  • For pgBackRest on GCS/S3/Azure Blob, provision object store infra as code.
    • GCS: org/gcp/pgbackrest-repo
    • S3: org/aws/pgbackrest-repo
    • Azure Blob: org/azure/pgbackrest-repo
  • Generate service account/API keys out-of-band and write them into the runtime vault bundle (hyops secrets set), not into Terraform variables/state.

Environment guardrails

Secret changes and rotations are expected to run behind environment guardrails to ensure:

  • target environment is explicit and validated
  • correlation identifiers exist across logs and artefacts

References: