Skip to content

Sync HashiCorp Vault secrets into runtime vault bundle

Purpose: Populate the runtime vault bundle from an external HashiCorp Vault authority for bootstrap, CI, and DR runs.
Owner: Platform operations
Trigger: Before running modules or runner-dispatched workflows that require secrets, or when rotating secrets in HashiCorp Vault.
Impact: Without this sync (or explicit overrides), modules may fail with missing env vars.
Severity: P3
Pre-reqs: Reachability to HashiCorp Vault, hyops init hashicorp-vault completed for the env, and ansible-vault installed when the token is cached in the runtime vault bundle.
Rollback strategy: Overwrite keys via hyops secrets set, or rotate in HashiCorp Vault and re-sync.

Context

This runbook deliberately distinguishes:

  • HashiCorp Vault = external secret authority
  • runtime vault bundle = per-environment encrypted cache at:
  • <root>/vault/bootstrap.vault.env

The runtime vault bundle is an execution cache, not the long-term source of truth.

The clean operator flow is:

  1. hyops init hashicorp-vault
  2. hyops secrets vault-sync
  3. optional hyops secrets set|ensure --persist vault

Secret mappings are driven by:

  • tools/secrets/hashicorp-vault/map/allowed.csv

Each mapped secret ref may include:

  • {env} placeholder
  • {scope} placeholder
  • optional #FIELD selector

Default resolution assumes a KV v2 engine unless overridden.

Preconditions and safety checks

  • Confirm Vault address and token source are available: 
    echo "$VAULT_ADDR"
echo "${VAULT_TOKEN:+set}"
    Notes:
  • after hyops init hashicorp-vault --persist-token, hyops secrets vault-sync can read the token from the runtime vault bundle

  • shell env still overrides the cached token when both exist

  • Confirm the env-scoped Vault init target is ready: 

    hyops init status --env dev

  • Confirm the vault password provider is ready (workstations): 

    hyops vault status-verbose
    Notes:

  • After hyops vault bootstrap, HyOps remembers the provider command at ~/.hybridops/config/vault-password-command.
  • In a fresh shell, if vault is locked, unlock once: 
    hyops vault password >/dev/null

Steps

1) Dry-run the sync

  • If this is a fresh environment, initialise Vault first: 

    VAULT_ADDR=https://vault.example.com \
VAULT_TOKEN=... \
hyops init hashicorp-vault --env dev

  • Command: 

    VAULT_ADDR=https://vault.example.com \
VAULT_TOKEN=... \
hyops secrets vault-sync --env dev --scope dr --dry-run

  • Expected result: a list of env keys that would be synced, with expanded Vault secret refs.

2) Run the sync

  • Command: 

    VAULT_ADDR=https://vault.example.com \
VAULT_TOKEN=... \
hyops secrets vault-sync --env dev --scope dr

  • Expected result: synced <N> keys into <root>/vault/bootstrap.vault.env.

3) Optional: use the same source during runner dispatch

  • Command: 
    VAULT_ADDR=https://vault.example.com \
VAULT_TOKEN=... \
hyops runner blueprint preflight --env dev \
  --runner-state-ref platform/linux/ops-runner#gcp_ops_runner_bootstrap \
  --secret-source vault \
  --secret-scope dr \
  --sync-env PATRONI_SUPERUSER_PASSWORD \
  --sync-env PATRONI_REPLICATION_PASSWORD \
  --sync-env PG_BACKUP_GCS_SA_JSON \
  --file "$HOME/.hybridops/envs/dev/config/blueprints/dr-postgresql-ha-failover-gcp.yml"

This refreshes the runtime vault bundle from HashiCorp Vault before staging the runner job.

Verification

  • Confirm the required env keys are present: 

    hyops secrets show --env dev \
  PATRONI_SUPERUSER_PASSWORD \
  PATRONI_REPLICATION_PASSWORD \
  PG_BACKUP_GCS_SA_JSON

  • Run module or runner preflight and confirm it no longer fails with missing env vars.

Troubleshooting

environment variable VAULT_TOKEN is required for Vault auth

  • Meaning: the selected token env var is not set in the current shell.

  • Remediation: 

    export VAULT_TOKEN='...'

failed to fetch Vault secret ...

  • Meaning: the secret ref, token policy, namespace, or engine/path resolution is wrong.

  • Remediation:

  • Confirm the mapped secret ref in tools/secrets/hashicorp-vault/map/allowed.csv
  • Confirm the KV engine mode:
    • --vault-engine kv-v2 (default)
    • --vault-engine kv-v1
  • Confirm the token policy can read the target path.

failed to read vault file ... unlock GPG in this shell

  • Meaning: the runtime vault cache exists but cannot be decrypted in the current shell.

  • Remediation: 

    hyops vault password >/dev/null

References

Maintainer: HybridOps.Studio
License: MIT-0 for code, CC-BY-4.0 for documentation