Sync GCP Secret Manager secrets into runtime vault bundle¶

Purpose: Populate the runtime vault bundle from GCP Secret Manager for bootstrap, CI, and runner-driven DR runs. Owner: Platform operations
Trigger: Before running GCP runner workflows that require secrets while on-prem or workstation-local secret sources are unavailable.
Impact: Without this sync (or explicit overrides), GCP runner-dispatched workflows may fail with missing env vars.
Severity: P3 Pre-reqs: gcloud and ansible-vault installed; access to the target GCP project; one of --project-id, --project-state-ref, env-scoped gcp.conf, or org/gcp/project-factory state available.
Rollback strategy: Overwrite keys via hyops secrets set, rotate in GCP Secret Manager, and re-sync.

Context¶

This runbook treats GCP Secret Manager as an external secret authority for the GCP DR lane.

HyOps syncs only an allowlisted set of keys into:

<root>/vault/bootstrap.vault.env

The runtime vault bundle remains the execution cache. GCP Secret Manager is the external authority for the selected secrets.

In the preferred GCP role split:

the runner and private DR VMs may live in the host/network project
GCP Secret Manager lives in the env control project

So it is normal for the runner to execute in one project while the synced secrets are read from another.

Default mapping file:

tools/secrets/gsm/map/allowed.csv

Preferred override path for site- or env-specific customization:

<root>/config/secrets/gsm/allowed.csv

Safe local template pattern:

<root>/config/secrets/gsm/allowed.csv.example

Project resolution order:

--project-id
--project-state-ref
GCP_PROJECT_ID
env-scoped gcp.conf
implicit org/gcp/project-factory state when available

Each mapped secret name may include:

{env}
{scope}
{env_key}
{env_key_slug}

Map resolution order:

--map-file
env-local runtime map: <root>/config/secrets/gsm/allowed.csv
HYOPS_GSM_MAP_FILE
shipped core default: tools/secrets/gsm/map/allowed.csv

The env-local runtime map augments or overrides the shipped core map. It does not replace the shipped default entirely. That keeps neutral product scopes such as dr, shared, and build available while still letting an env add private integration rows such as Academy or platform IdP secrets.

When you need an env-local override, keep a commented starter file such as allowed.csv.example beside the live map and only copy it to allowed.csv when you deliberately want this env to diverge from the shipped default.

You can also register missing env-local rows directly from the secret-write path:

hyops secrets set --env dev \
  --persist gsm \
  --persist-scope academy \
  --persist-register-gsm-map \
  LEARN_SESSION_SECRET=... \
  ENTITLEMENTS_API_TOKEN=...

That writes missing rows only into:

<root>/config/secrets/gsm/allowed.csv

It does not modify the shipped core default map.

Before those rows are written, HyOps validates that the target GCP project can actually be used for Secret Manager persistence. If GCP access fails, the command exits without changing the env-local GSM map.

Preconditions and safety checks¶

Confirm the active GCP account can read secrets:

gcloud auth list
Confirm the env has GCP init data or provide a project explicitly:

hyops show init --env dev gcp
Confirm the vault password provider is ready (workstations):

hyops vault status-verbose

If vault is locked in this shell:

  hyops vault password >/dev/null

Steps¶

1) Dry-run the sync

Command:

hyops secrets gsm-sync --env dev --scope dr --dry-run
Expected result: a list of env keys that would be synced, with the resolved GCP project id and mapped secret names.

2) Run the sync

Command:

hyops secrets gsm-sync --env dev --scope dr
Expected result: synced <N> keys into <root>/vault/bootstrap.vault.env.

3) Use GCP Secret Manager during runner dispatch

Command:

hyops runner blueprint preflight --env dev \ --runner-state-ref platform/linux/ops-runner#gcp_ops_runner_bootstrap \ --secret-source gsm \ --secret-scope dr \ --gsm-project-state-ref org/gcp/project-factory \ --sync-env PATRONI_SUPERUSER_PASSWORD \ --sync-env PATRONI_REPLICATION_PASSWORD \ --sync-env PG_BACKUP_GCS_SA_JSON \ --file "$HOME/.hybridops/envs/dev/config/blueprints/dr-postgresql-ha-failover-gcp.yml"

This refreshes the runtime vault bundle from GCP Secret Manager before the runner job is staged. For the preferred layout, org/gcp/project-factory should be treated as the env control project rather than the host/network project that owns the runner VM.

4) Optional: seed GCP Secret Manager from the current runtime vault bundle

Command:

hyops secrets gsm-persist --env dev --scope dr
Expected result: the allowlisted DR keys currently stored in <root>/vault/bootstrap.vault.env are written to GCP Secret Manager, creating the secret if needed and adding a new version when it already exists.

5) Optional: seed build-time publish credentials for the VyOS record path

Command:

hyops secrets gsm-persist --env dev --scope build
Expected result: HYOPS_VYOS_GCS_SA_JSON is written to GCP Secret Manager using the allowlisted build-scope secret name, so runner-driven image builds can publish without depending on a local shell export.

6) Optional: seed env-local private scopes

Private Academy, Moodle, and external identity-provider mappings should be defined in the env-local override map, not in the shipped core default:

<root>/config/secrets/gsm/allowed.csv

Example:

cp <root>/config/secrets/gsm/allowed.csv.example \
  <root>/config/secrets/gsm/allowed.csv
hyops secrets gsm-persist --env dev --scope academy

Expected result: the env-local Academy allowlisted keys are written to GCP Secret Manager so External Secrets can project them into the burst cluster.

Verification¶

Confirm the required env keys are present:

hyops secrets show --env dev \ PATRONI_SUPERUSER_PASSWORD \ PATRONI_REPLICATION_PASSWORD \ PG_BACKUP_GCS_SA_JSON
Re-run the affected module or runner preflight and confirm it no longer fails with missing env vars.

Troubleshooting¶

`GCP project id is not configured`¶

Meaning: HyOps could not resolve the Secret Manager project from --project-id, --project-state-ref, GCP_PROJECT_ID, the env-scoped gcp.conf, or implicit org/gcp/project-factory state.
Remediation:

hyops init gcp --env dev --with-cli-login --force

or:

  hyops secrets gsm-sync --env dev --project-id <project-id>

or:

  hyops secrets gsm-sync --env dev --project-state-ref org/gcp/project-factory

`failed to fetch GCP Secret Manager secret`¶

Meaning: the secret name is wrong, the project is wrong, or the current GCP identity cannot read the secret.
Remediation:
confirm the mapped secret name in tools/secrets/gsm/map/allowed.csv or the env-local override at <root>/config/secrets/gsm/allowed.csv
confirm the secret exists in the resolved project
confirm gcloud auth is using an identity with secretAccessor

`Secret Manager API is not enabled on project ...`¶

Meaning: secretmanager.googleapis.com is disabled in the resolved GCP project.
Remediation:

gcloud services enable secretmanager.googleapis.com --project

or enable it declaratively in org/gcp/project-factory:

  activate_apis:
    - secretmanager.googleapis.com

then re-apply the project-factory module.

`failed to read vault file ... unlock GPG in this shell`¶

Meaning: the runtime vault bundle exists but cannot be decrypted in the current shell.
Remediation:

hyops vault password >/dev/null

References¶

License: MIT-0 for code, CC-BY-4.0 for documentation