Skip to content

Sync GCP Secret Manager secrets into runtime vault bundle

Purpose: Populate the runtime vault bundle from GCP Secret Manager for bootstrap, CI, and runner-driven DR runs.
Owner: Platform operations
Trigger: Before running GCP runner workflows that require secrets while on-prem or workstation-local secret sources are unavailable.
Impact: Without this sync (or explicit overrides), GCP runner-dispatched workflows may fail with missing env vars.
Severity: P3
Pre-reqs: gcloud and ansible-vault installed; access to the target GCP project; one of --project-id, --project-state-ref, env-scoped gcp.conf, or org/gcp/project-factory state available.
Rollback strategy: Overwrite keys via hyops secrets set, rotate in GCP Secret Manager, and re-sync.

Context

This runbook treats GCP Secret Manager as an external secret authority for the GCP DR lane.

HyOps syncs only an allowlisted set of keys into:

  • <root>/vault/bootstrap.vault.env

The runtime vault bundle remains the execution cache. GCP Secret Manager is the external authority for the selected secrets.

In the preferred GCP role split:

  • the runner and private DR VMs may live in the host/network project
  • GCP Secret Manager lives in the env control project

So it is normal for the runner to execute in one project while the synced secrets are read from another.

Default mapping file:

  • tools/secrets/gsm/map/allowed.csv

Project resolution order:

  1. --project-id
  2. --project-state-ref
  3. implicit org/gcp/project-factory state when available
  4. GCP_PROJECT_ID
  5. env-scoped gcp.conf

Each mapped secret name may include:

  • {env}
  • {scope}

Preconditions and safety checks

  • Confirm the active GCP account can read secrets: 

    gcloud auth list

  • Confirm the env has GCP init data or provide a project explicitly: 

    hyops init status --env dev

  • Confirm the vault password provider is ready (workstations): 

    hyops vault status-verbose
    If vault is locked in this shell: 
    hyops vault password >/dev/null

Steps

1) Dry-run the sync

  • Command: 

    hyops secrets gsm-sync --env dev --scope dr --dry-run

  • Expected result: a list of env keys that would be synced, with the resolved GCP project id and mapped secret names.

2) Run the sync

  • Command: 

    hyops secrets gsm-sync --env dev --scope dr

  • Expected result: synced <N> keys into <root>/vault/bootstrap.vault.env.

3) Use GCP Secret Manager during runner dispatch

  • Command: 
    hyops runner blueprint preflight --env dev \
  --runner-state-ref platform/linux/ops-runner#gcp_ops_runner_bootstrap \
  --secret-source gsm \
  --secret-scope dr \
  --gsm-project-state-ref org/gcp/project-factory \
  --sync-env PATRONI_SUPERUSER_PASSWORD \
  --sync-env PATRONI_REPLICATION_PASSWORD \
  --sync-env PG_BACKUP_GCS_SA_JSON \
  --file "$HOME/.hybridops/envs/dev/config/blueprints/dr-postgresql-ha-failover-gcp.yml"

This refreshes the runtime vault bundle from GCP Secret Manager before the runner job is staged. For the preferred layout, org/gcp/project-factory should be treated as the env control project rather than the host/network project that owns the runner VM.

4) Optional: seed GCP Secret Manager from the current runtime vault bundle

  • Command: 

    hyops secrets gsm-persist --env dev --scope dr

  • Expected result: the allowlisted DR keys currently stored in <root>/vault/bootstrap.vault.env are written to GCP Secret Manager, creating the secret if needed and adding a new version when it already exists.

5) Optional: seed build-time publish credentials for the VyOS artifact path

  • Command: 

    hyops secrets gsm-persist --env dev --scope build

  • Expected result: HYOPS_VYOS_GCS_SA_JSON is written to GCP Secret Manager using the allowlisted build-scope secret name, so runner-driven image builds can publish without depending on a local shell export.

Verification

  • Confirm the required env keys are present: 

    hyops secrets show --env dev \
  PATRONI_SUPERUSER_PASSWORD \
  PATRONI_REPLICATION_PASSWORD \
  PG_BACKUP_GCS_SA_JSON

  • Re-run the affected module or runner preflight and confirm it no longer fails with missing env vars.

Troubleshooting

GCP project id is not configured

  • Meaning: HyOps could not resolve the Secret Manager project from --project-id, --project-state-ref, GCP_PROJECT_ID, the env-scoped gcp.conf, or implicit org/gcp/project-factory state.

  • Remediation: 

    hyops init gcp --env dev --force
    or: 
    hyops secrets gsm-sync --env dev --project-id <project-id>
    or: 
    hyops secrets gsm-sync --env dev --project-state-ref org/gcp/project-factory

failed to fetch GCP Secret Manager secret

  • Meaning: the secret name is wrong, the project is wrong, or the current GCP identity cannot read the secret.

  • Remediation:

  • confirm the mapped secret name in tools/secrets/gsm/map/allowed.csv
  • confirm the secret exists in the resolved project
  • confirm gcloud auth is using an identity with secretAccessor

Secret Manager API is not enabled on project ...

  • Meaning: secretmanager.googleapis.com is disabled in the resolved GCP project.

  • Remediation: 

    gcloud services enable secretmanager.googleapis.com --project <project-id>

or enable it declaratively in org/gcp/project-factory:

activate_apis:
  - secretmanager.googleapis.com

then re-apply the project-factory module.

failed to read vault file ... unlock GPG in this shell

  • Meaning: the runtime vault bundle exists but cannot be decrypted in the current shell.

  • Remediation: 

    hyops vault password >/dev/null

References

Maintainer: HybridOps.Studio
License: MIT-0 for code, CC-BY-4.0 for documentation