Skip to content

Sync HashiCorp Vault secrets into runtime vault bundle

  • Purpose: Populate the runtime vault bundle from an external HashiCorp Vault authority for bootstrap, CI, and DR runs. Owner: Platform operations

  • Trigger: Before running modules or runner-dispatched workflows that require secrets, or when rotating secrets in HashiCorp Vault.

  • Impact: Without this sync (or explicit overrides), modules may fail with missing env vars.

  • Severity: P3 Pre-reqs: Reachability to HashiCorp Vault, hyops init hashicorp-vault completed for the env, and ansible-vault installed when the token is cached in the runtime vault bundle.

  • Rollback strategy: Overwrite keys via hyops secrets set, or rotate in HashiCorp Vault and re-sync.

Context

This runbook deliberately distinguishes:

  • HashiCorp Vault = external secret authority
  • runtime vault bundle = per-environment encrypted cache at:
  • <root>/vault/bootstrap.vault.env

The runtime vault bundle is an execution cache, not the long-term source of truth.

The clean operator flow is:

  1. hyops init hashicorp-vault
  2. hyops secrets vault-sync
  3. optional hyops secrets set|ensure --persist vault

Secret mappings are driven by:

  • tools/secrets/hashicorp-vault/map/allowed.csv

Each mapped secret ref may include:

  • {env} placeholder
  • {scope} placeholder
  • optional #FIELD selector

Default resolution assumes a KV v2 engine unless overridden.

Preconditions and safety checks

  • Confirm Vault address and token source are available:

    echo "$VAULT_ADDR" echo "${VAULT_TOKEN:+set}"

Notes: - after hyops init hashicorp-vault --persist-token, hyops secrets vault-sync can read the token from the runtime vault bundle - shell env still overrides the cached token when both exist

  • Confirm the env-scoped Vault init target is ready:

    hyops init status --env dev

  • Confirm the vault password provider is ready (workstations):

    hyops vault status-verbose

Notes: - After hyops vault bootstrap, HyOps remembers the provider command at ~/.hybridops/config/vault-password-command. - In a fresh shell, if vault is locked, unlock once:

    hyops vault password >/dev/null

Steps

1) Dry-run the sync

  • If this is a fresh environment, initialise Vault first:

    VAULT_ADDR=https://vault.example.com \ VAULT_TOKEN=... \ hyops init hashicorp-vault --env dev

  • Command:

    VAULT_ADDR=https://vault.example.com \ VAULT_TOKEN=... \ hyops secrets vault-sync --env dev --scope dr --dry-run

  • Expected result: a list of env keys that would be synced, with expanded Vault secret refs.

2) Run the sync

  • Command:

    VAULT_ADDR=https://vault.example.com \ VAULT_TOKEN=... \ hyops secrets vault-sync --env dev --scope dr

  • Expected result: synced <N> keys into <root>/vault/bootstrap.vault.env.

3) Optional: use the same source during runner dispatch

  • Command:

    VAULT_ADDR=https://vault.example.com \ VAULT_TOKEN=... \ hyops runner blueprint preflight --env dev \ --runner-state-ref platform/linux/ops-runner#gcp_ops_runner_bootstrap \ --secret-source vault \ --secret-scope dr \ --sync-env PATRONI_SUPERUSER_PASSWORD \ --sync-env PATRONI_REPLICATION_PASSWORD \ --sync-env PG_BACKUP_GCS_SA_JSON \ --file "$HOME/.hybridops/envs/dev/config/blueprints/dr-postgresql-ha-failover-gcp.yml"

This refreshes the runtime vault bundle from HashiCorp Vault before staging the runner job.

Verification

  • Confirm the required env keys are present:

    hyops secrets show --env dev \ PATRONI_SUPERUSER_PASSWORD \ PATRONI_REPLICATION_PASSWORD \ PG_BACKUP_GCS_SA_JSON

  • Run module or runner preflight and confirm it no longer fails with missing env vars.

Troubleshooting

environment variable VAULT_TOKEN is required for Vault auth

  • Meaning: the selected token env var is not set in the current shell.

  • Remediation:

    export VAULT_TOKEN='...'

failed to fetch Vault secret ...

  • Meaning: the secret ref, token policy, namespace, or engine/path resolution is wrong.

  • Remediation:

  • Confirm the mapped secret ref in tools/secrets/hashicorp-vault/map/allowed.csv
  • Confirm the KV engine mode:
    • --vault-engine kv-v2 (default)
    • --vault-engine kv-v1
  • Confirm the token policy can read the target path.

failed to read vault file ... unlock GPG in this shell

  • Meaning: the runtime vault cache exists but cannot be decrypted in the current shell.

  • Remediation:

    hyops vault password >/dev/null

References

License: MIT-0 for code, CC-BY-4.0 for documentation