Skip to content

Secrets lifecycle and responsibilities

Defines how secrets are handled from bootstrap through steady state and disaster recovery.

Design references:

Goals

  • Keep shipped HybridOps defaults neutral and product-safe.
  • Separate bootstrap connectivity, runtime execution secrets, and steady-state application secret authorities.
  • Keep business- or tenant-specific secret naming out of the shipped core release.
  • No plaintext secret values committed to Git.
  • No hardcoded credentials in playbooks, pipelines, templates, or images.

Runtime root and environments

HybridOps stores local runtime records under a runtime root (<root>), including credentials and the encrypted vault bundle.

<root> resolves to:

  • ~/.hybridops/envs/<env> when you pass --env <env> (recommended)
  • ~/.hybridops when no environment is specified (legacy single-env mode)

Common paths used by this guide:

  • Config: <root>/config/
  • Credentials: <root>/credentials/
  • Vault bundle: <root>/vault/bootstrap.vault.env

Reference:

Secret classes

Bootstrap connectivity secrets

Secrets used to allow operators and automation runners (the execution agents that run platform operations within a site boundary) to authenticate to infrastructure control planes.

Primary records (generated by hyops init <target>):

  • Config template: <root>/config/<target>.conf
  • Runtime credentials: <root>/credentials/<target>.credentials.tfvars (local-only, mode 0600)
  • Readiness marker: <root>/meta/<target>.ready.json (non-secret)

Constraints:

  • Not committed to Git.
  • Minimal scope: endpoints and credentials required to reach provider APIs.
  • Treated as sensitive local state (mode 0600).

Encrypted vault bundle

Encrypted vault bundle used to support non-interactive automation for bootstrap, CI, and DR.

  • File: <root>/vault/bootstrap.vault.env (ansible-vault encrypted env-format file)
  • Scope: bootstrap-only secrets and early pipeline credentials where required.
  • Vault password: stored out-of-band and injected at runtime (for example via CI secret or a local password provider).

Local workstation provider (recommended):

  • hyops vault bootstrap stores the vault password in pass (default entry: hybridops/ansible-vault).
  • The provider script is tools/secrets/vault/vault-pass.sh in the HybridOps.Core release root. It is discovered automatically when you run HybridOps from the core repo, or can be overridden via hyops vault --script <path>.
  • After successful bootstrap, HybridOps persists the default provider command at ~/.hybridops/config/vault-password-command so other HybridOps commands can decrypt vault files without extra flags.
  • In a fresh shell, unlock your GPG key once before non-interactive operations:
  • hyops vault password >/dev/null
  • If a command reports vault password command timed out while waiting for unlock, unlock once and optionally increase timeout:
  • hyops vault password >/dev/null
  • export HYOPS_VAULT_PASSWORD_COMMAND_TIMEOUT_S=300

Seeding secrets into the vault bundle:

  • Generate random defaults (bootstrap/labs/CI/DR):
  • hyops secrets ensure --env <env> KEY1 KEY2 ...
  • hyops secrets ensure --env <env> --module <module_ref> --inputs <inputs.yml>
  • Rotate existing keys: hyops secrets ensure --env <env> --force KEY1 [KEY2 ...]
  • Manual set (explicit overrides):
  • hyops secrets set --env <env> KEY=VALUE ...
  • hyops secrets set --env <env> --from-env KEY1 KEY2 ...
  • hyops secrets set --env <env> --from-file KEY=PATH ...
  • Enterprise sync (AKV -> vault bundle):
  • hyops secrets akv-sync --env <env> --vault-name <name> [--scope <scope>]

Notes:

  • The vault bundle is not a steady-state secrets store. It is a bootstrap/CI/DR convenience.
  • HybridOps module state MUST NOT publish secret values. Modules publish env-var references (for example db_password_env) instead.

Constraints:

  • Plaintext derivatives must not be committed and must not persist beyond a single command execution.
  • The bundle is not a steady-state secret store.
  • Use --from-file for multiline secrets such as private keys, certificates, and JSON credentials instead of inline KEY=VALUE shell quoting.

Validation and failure semantics (fail-fast)

HybridOps validates secrets in two layers:

  1. Module input validation (static)
  2. Modules declare which env keys they require via inputs.required_env.
  3. Any *_env fields (for example db_password_env, secret_key_env) MUST reference keys included in inputs.required_env.

  4. Validators fail before any provisioning work is executed if the contract is violated.

  5. Driver validation (runtime)

  6. Drivers validate that required env vars exist with non-empty values.
  7. The Ansible driver can optionally load <root>/vault/bootstrap.vault.env (when inputs.load_vault_env=true or when required keys are missing in the shell env).
  8. If required keys are still missing after vault merge, the command fails before running Ansible.

Recommended workflow:

  • Gate every config module run with:
  • hyops preflight --env <env> --strict --module <module_ref> --inputs <inputs.yml>

Typical failure messages are explicit:

  • Missing vault password provider:
  • [vault-pass] ERROR: entry not ready (run --bootstrap)
  • Missing required env vars:
  • missing required env vars: ... Generate random defaults: hyops secrets ensure --env <env> ... Or set explicit values: hyops secrets set --env <env> KEY=VALUE ...

Platform and application secrets

Secrets used by platform services and workloads.

Authoritative source depends on the lane and provider:

  • Azure-backed lanes may use Azure Key Vault.
  • GCP-backed lanes may use GCP Secret Manager.
  • Some bootstrap and DR flows intentionally use the encrypted runtime vault bundle as the canonical recovery cache until a cloud secret authority is rehydrated.

Access patterns:

  • Platform tooling reads secrets through explicit sync or persist workflows.
  • Kubernetes consumes secrets through External Secrets Operator or equivalent provider-native projection.
  • Runtime execution can use the encrypted vault bundle as a non-interactive cache for bootstrap, CI, DR, and runner-driven operations.

Constraints:

  • Git stores references and wiring, not secret values.
  • Kubernetes Secret objects are runtime projections and not primary storage.
  • The shipped core allowlist must stay generic; site- or business-specific scopes belong in env-local overrides.

Allowlisted external secret mapping

HybridOps ships a narrow default GSM allowlist in core:

  • tools/secrets/gsm/map/allowed.csv

That shipped map should contain only product-default scopes such as:

  • shared
  • dr
  • build

Site- or business-specific mappings must live in the env runtime, not in core:

  • <root>/config/secrets/gsm/allowed.csv

Use the local starter file when an env needs its own mapping contract:

  • <root>/config/secrets/gsm/allowed.csv.example

This keeps the tarball neutral while still allowing private Academy, Moodle, or identity-provider naming per env.

Optional break-glass record

Optional DR record used only when required by DR policy or when the vault bundle scope is intentionally insufficient for recovery.

Example location:

  • docs/secrets/secrets.dr.enc.yaml

Constraints:

  • Minimal set required for the intended break-glass scenario.
  • Encrypted with SOPS.
  • Used only under a documented DR runbook.

Lifecycle by phase

Bootstrap

  1. Install the runtime and prerequisites:
  2. hyops setup all --sudo
  3. Bootstrap the vault password provider (workstations) or provide a CI password source:
  4. Workstation: hyops vault bootstrap
  5. Validate/unlock in current shell: hyops vault status-verbose then hyops vault password >/dev/null if needed
  6. CI: provide an Ansible Vault password source (for example ANSIBLE_VAULT_PASSWORD_FILE or --vault-password-command)
  7. Initialise provider access:
  8. hyops init <target> writes bootstrap config/credentials under <root>/config/ and <root>/credentials/
  9. Seed bootstrap-only secrets if required:
  10. Generate defaults: hyops secrets ensure --env <env> KEY1 KEY2 ...
  11. Override explicitly: hyops secrets set --env <env> KEY=VALUE ...
  12. Enterprise sync (AKV -> vault bundle): hyops secrets akv-sync --env <env> ...
  13. Use bootstrap connectivity for early provisioning:
  14. create foundations, establish AKV, and seed initial platform secrets.

Security intent:

  • Bootstrap records remain minimal and local-only.
  • Secret values move to AKV as early as practical.

Steady state

  • The active secret authority is the provider or secret system selected for that lane.
  • External Secrets Operator projects those values into Kubernetes where required.
  • Bootstrap connectivity records remain limited to infrastructure access and are not used as a general secret store.
  • The runtime vault bundle remains available for controlled bootstrap, CI, and DR use, but it is not a substitute for a steady-state secret authority.

Disaster recovery

  • DR automation is executed from an external runner using the encrypted vault bundle for non-interactive secret injection.
  • Typical actions: provision or scale target clusters, promote database replicas, perform DNS cutover.
  • Optional break-glass record is reserved for explicitly documented scenarios.

Design principles

  • Git stores references and mapping contracts, not secret values.
  • Encrypted vault bundles are permitted for bootstrap, CI, and DR only.
  • Bootstrap records must not become an unmanaged second live secret system.
  • Provider secret authorities should be rehydrated from the runtime vault when rebuilding a lane.
  • Shipped core secret maps must stay neutral; env-local overrides hold private naming.
  • Cloud backup identities must avoid key material in Terraform state.
  • For pgBackRest on GCS/S3/Azure Blob, provision object store infra as code.
    • GCS: org/gcp/pgbackrest-repo
    • S3: org/aws/pgbackrest-repo
    • Azure Blob: org/azure/pgbackrest-repo
  • Generate service account/API keys out-of-band and write them into the runtime vault bundle (hyops secrets set), not into Terraform variables/state.

Environment guardrails

Secret changes and rotations are expected to run behind environment guardrails to ensure:

  • target environment is explicit and validated
  • correlation identifiers exist across logs and records

References: